How to assess · For hiring teams

How to Assess Test Automation Skills When Hiring

The test formats that actually work for Test Automation, what a strong answer looks like, sample questions and a scoring rubric you can use as-is.

The short answer

Assess Test Automation with a task, not a conversation: test a small feature end to end, fix a flaky test suite, ai-scored assessment (e.g. cohesyve) or review a pull request of tests. Score it against written criteria you fix before you see any submissions, and weight the criteria that the role actually depends on.

Chooses the right level for each check — unit, integration, end-to-end — and can say why most tests should not be end-to-end
Writes tests that fail for one reason and say what it is
Makes tests deterministic: controls time, randomness, network and data
Designs page objects, fixtures and helpers so that a UI change touches one place

Build a Test Automation assessment →Read the full article ↓

Paste a job description; Cohesyve generates a role-specific assessment and rubric. Ten candidates free, no card.

Test automation is a skill that hides in plain sight. Almost every engineer has written tests; far fewer can build a suite that the team trusts — one that catches real regressions, runs fast enough to be run, fails for real reasons, and does not need a week of maintenance every time the UI changes. That is what you are hiring for when you hire a QA automation engineer or an SDET, and it does not show on a list of frameworks. This page covers how to assess test automation as engineering: what to test at which level, how to make tests deterministic, how to design for maintainability, and how to tell a flaky suite from a broken product.

Why Test Automation is worth testing

A bad automation suite is worse than none. It is slow, so people stop running it; it is flaky, so people stop trusting it; it tests implementation details, so every refactor breaks it. The team ends up maintaining tests that catch nothing while real bugs ship. Testing reveals whether a candidate designs for signal and durability, or writes tests that pass today and cost the team for years.

What strong Test Automation looks like

Chooses the right level for each check — unit, integration, end-to-end — and can say why most tests should not be end-to-end
Writes tests that fail for one reason and say what it is
Makes tests deterministic: controls time, randomness, network and data
Designs page objects, fixtures and helpers so that a UI change touches one place
Treats flakiness as a bug to be fixed at the cause, not retried until green
Knows what to leave untested and can defend the choice
Measures the suite — duration, flake rate, coverage of risk — and improves it

Ways to assess Test Automation

Test a small feature end to end

Provide a small application with a form, an API and a database. Ask for a test plan that assigns checks to levels, plus implementation of two or three tests at different levels. Ninety minutes or a capped take-home.

Pros

Shows the level judgement and the implementation quality together.

Cons

Setup cost; framework familiarity affects speed, so let them choose tools.

Best for SDET and automation engineer roles at any level.

Fix a flaky test suite

Give a suite with three sources of flakiness — a sleep-based wait, a shared mutable fixture, a real network call — and ask them to diagnose and fix without weakening the tests.

Pros

Tests the diagnostic skill that determines whether a suite stays trusted; hard to bluff.

Cons

Needs a well-crafted fixture in a framework they know.

Best for Mid and senior roles inheriting an existing suite.

AI-scored assessment (e.g. Cohesyve)

Generate a test-design task from the job description — a test plan for a described feature, a flaky-test diagnosis, a review of a suite — with a rubric. Each candidate receives a different variant; written reasoning is scored.

Pros

Asynchronous and consistent; a different feature per candidate; test design is mostly reasoning, which scores well.

Cons

Cannot run their tests; confirm implementation with finalists.

Best for Screening before hands-on interviews.

Review a pull request of tests

Provide tests that assert on implementation details, share state, and have unclear names. Ask for review comments.

Pros

Fast; reveals taste and the ability to raise the bar for others.

Cons

Passive; pair with something hands-on.

Best for Senior and lead roles.

Cohesyve

Run a Test Automation assessment on your next opening

Cohesyve generates a unique Test Automation task per candidate from your job description, with the scoring rubric attached. Questions are different for every applicant, so they cannot be shared or looked up.

Try it free →Book a 30-min demo

What to test

Test strategy

Whether they put checks at the right level.

Assign twelve checks for a checkout flow to unit, integration or end-to-end and justify eachExplain why an end-to-end test for a validation rule is the wrong choiceDecide what not to automate for a feature that changes weekly

Determinism and isolation

Whether tests are reliable by design.

Remove a `sleep` and replace it with a proper waitMake a test independent of the clockIsolate a test that depends on another test's data

Maintainability

Whether the suite survives change.

Introduce a page object so a selector change is a one-line fixRefactor duplicated setup into a fixtureName tests so a failure explains itself

Suite health

Whether they manage the suite as a product.

Propose a policy for flaky testsCut a forty-minute suite to ten without losing signalChoose metrics for suite quality and say what they would do with them

Sample Test Automation questions

What is the difference between a unit test and an integration test, and when does each earn its place?

Entry

Look for Isolation and speed versus realism; most logic unit-tested, boundaries integration-tested, a few flows end to end.

This test uses `sleep(3)` before asserting. What is wrong, and what do you do?

Entry

Look for It is both slow and flaky; replace with an explicit wait on the condition.

A test passes locally and fails in CI one time in ten. How do you approach it?

Mid

Look for Reproduce with repetition, look for shared state, timing, environment differences; fix the cause; do not add retries as the fix.

Design the test approach for a payments feature with a third-party provider.

Mid

Look for Contract tests against the provider's API, a fake for most tests, a small number of real sandbox tests, clear ownership of the fake.

Your end-to-end suite takes an hour and developers have stopped running it. What do you do?

Senior

Look for Push checks down the pyramid, parallelise, shard, remove redundant tests, run a smoke subset on every change and the rest nightly, and measure.

Red flags

Every test is end-to-end through the UI
Fixes flakiness with retries
Tests assert on internal implementation rather than behaviour
Cannot say how long the last suite they owned took or how often it flaked
Sees automation as recording clicks rather than engineering

Scoring rubric

Criterion	Weight	What strong looks like
Strategy and level judgement	30%	Checks live at the cheapest level that gives real signal.
Determinism	25%	Time, data, network and order are controlled; flakes are fixed at the cause.
Maintainability	20%	Changes touch one place; failures explain themselves.
Implementation quality	15%	Tests are readable, focused and fast.
Suite ownership	10%	Measures and improves the suite as a product.

Mistakes hiring teams make

Testing framework syntax rather than test design
Not including a flaky-test scenario — it is the defining skill
Accepting a suite of only end-to-end tests because it "covers everything"
Ignoring how long the tests take to run
Hiring manual QA experience for an automation role without testing engineering skill

Roles that need Test Automation

QA Automation EngineerSDETTest EngineerSoftware EngineerQuality EngineerFrontend Developer

Common questions

Should I test a specific framework like Playwright or Selenium?

Let candidates use what they know unless the role is locked to one tool. Test design and determinism transfer between frameworks; syntax does not predict either.

How is an SDET different from a QA engineer?

An SDET builds test infrastructure and suites as software; a QA engineer may focus on test design and manual verification. Assess an SDET as an engineer: design, code quality, and how they handle flakiness.

What is a reasonable amount of end-to-end testing?

A small number of critical flows — enough to catch a broken build, not enough to verify every rule. Most checks belong lower where they are faster and more reliable.

How do I assess test automation without a codebase to give?

A described feature and a test-plan exercise, plus a flaky-test scenario on paper, reveal most of the judgement. Implementation can be confirmed with finalists on a small fixture.

Cohesyve · Skill assessments for hiring

Test Test Automation before the first interview

Generate a role-specific Test Automation assessment from your job description and see who can do the work before you spend interview time on them.

1,500+

assessments completed

50%

faster time-to-hire

90%

completion rate

5 min

from JD to assessment

No credit card · 10 free candidates · Plans sized to your hiring volume

From the blog

Related skill guides