Skip to content
>_ StartupJobList_

Assess the work

Software QA interview scorecard: release confidence under retries

Assess how a candidate prioritizes product risk, defines a trustworthy oracle, isolates test data, diagnoses contradictory runs and states what the evidence cannot prove. The exercise is tool-agnostic and does not claim to validate hiring outcomes.

Run a bounded 60-minute discussion

This is a structured discussion about supplied evidence, not a coding-speed test, Playwright syntax quiz or validated hiring instrument. Share the full scenario, timing and adjustment path in advance. Ask the five core prompts in order and score each only against its named competency. Optional discussion never replaces a missed core prompt.

Fixed agenda. Keep a visible time boundary so each candidate gets the same core opportunity.
TimeStageAction
0–5OrientExplain the format, adjustment route and synthetic facts; answer process questions without adding scenario facts.
5–12Core prompt 1Risk prioritization.
12–20Core prompt 2Oracle quality.
20–28Core prompt 3Test isolation.
28–38Core prompt 4Failure diagnosis.
38–48Core prompt 5Evidence limits and release authority.
48–55Candidate questionsAnswer consistently; do not introduce facts that change the scored scenario.
55–60CloseExplain next steps and note any process gap. Do not add a sixth scored competency.

Stop rule: move on at each time boundary. If a prompt is not fairly elicited, mark only its competency NO and use the same follow-up opportunity for candidates in the stage. Do not make up observations or score a tool name.

Playwright's documentation describes the runner's locator/actionability and fixture behavior. It does not establish that this scorecard is predictive or that Playwright is required for every QA role: actionability and fixtures.

↑ Back to sections

Give the candidate the complete scenario and prompts

This packet stands alone when printed. It contains the same facts and five timed core prompts as the interviewer packet, without scoring guidance.

The candidate artifact can be printed on its own.

This is a 60-minute discussion about test evidence, not a coding-speed test. Use only the supplied synthetic facts. You may ask for a fact to be repeated, explain an assumption or identify an owner who must provide missing information. No specific browser tool is required.

Scenario: invitation acceptance in workspace W-31

A user accepts invitation I-17 while a workspace has one seat left. The product contract says an invitation is “accepted” only after the service confirms exactly one durable membership and reserves that seat. While work is pending, the interface must say so. The candidate is assessing release evidence, not being asked to implement the feature.

All facts below are synthetic and supplied for this exercise. Do not infer facts that are not listed.
EvidenceObservation
API contract noteThe acceptance endpoint changed from HTTP 201 to HTTP 202. A 202 returns an operation URL. The operation can be PENDING before it becomes SUCCEEDED. Requests must carry an Idempotency-Key; repeated requests with the same key resolve to one operation. Requests without a key are not guaranteed to deduplicate.
Unit run42 of 42 tests pass. The acceptance client is stubbed to return HTTP 201 with status: accepted. No service or persisted membership is exercised.
Browser run PW-128 of 8 tests pass. The POST is intercepted and returns HTTP 201. The test clicks once and checks that an “Invitation accepted” toast appears. It does not reload or read durable membership state.
Browser run PW-18Against the updated service, the POST returns HTTP 202 and an operation URL. The operation remains PENDING for 1.1 seconds, then becomes SUCCEEDED. The current interface shows “Invitation accepted” as soon as it receives any 2xx response. The test expecting HTTP 201 fails.
Parallel run PW-21Two workers use the shared workspace fixture W-31 and invitation I-17. Worker B cleanup deletes I-17 while worker A polls its operation. Two of ten first attempts return 404; both pass on retry.
Request trace R-44During one delayed acceptance, the user clicks twice 400 ms apart. The browser sends two POST requests, both without an Idempotency-Key, and the service logs two operation IDs. The trace does not show whether either request created a durable membership.

There is no production or customer data in this exercise. No browser runner, source code, database access or specific testing tool is required. You may ask for a stated fact to be repeated, explain an assumption, or say which owner must supply missing information.

Five timed questions

  1. 5–12 minutes: Which observations most change your release risk, what would you investigate first, and what fact or owner do you need next?
  2. 12–20 minutes: What would a passing check need to observe for this workflow, and where would you observe it? Which current checks would you keep or change?
  3. 20–28 minutes: How would you make parallel runs reproducible without letting one run's setup or cleanup affect another?
  4. 28–38 minutes: Walk through how you would investigate the first failure from the supplied evidence. What would you capture next, and how would you preserve the first result?
  5. 38–48 minutes: Given these observations, what can the team conclude and what remains unknown? Who should decide whether to release, and what evidence should inform that decision?

↑ Back to sections

Prompt 1: prioritize the risk

5–12 minutes: “Which observations most change your release risk, what would you investigate first, and what fact or owner do you need next?”

Listen for user impact, at-most-once membership/seat behavior and honest pending state. The exercise does not supply an organizational severity scale or release policy; a candidate can make a bounded recommendation while naming those unknowns.

↑ Back to sections

Prompt 2: define a useful oracle

12–20 minutes: “What would a passing check need to observe for this workflow, and where would you observe it? Which current checks would you keep or change?”

Look for the distinction between an HTTP acknowledgment, durable membership/seat state and user-facing status. A check need not use Playwright if another layer observes the same contract with a clearer signal.

↑ Back to sections

Prompts 3 and 4: control data and diagnose failure

20–28 minutes: “How would you make parallel runs reproducible without letting one run's setup or cleanup affect another?”

Look for bounded per-test or per-worker ownership, safe cleanup and an understanding of the shared W-31/I-17 collision. Turning off parallelism may be a temporary containment measure, but by itself does not explain ownership or prove the product path.

28–38 minutes: “Walk through how you would investigate the first failure from the supplied evidence. What would you capture next, and how would you preserve the first result?”

Look for correlation across request/operation IDs, response state, UI state and fixture lifecycle. A retry can reveal instability; a retry-to-green does not erase the initial failure or identify its cause.

↑ Back to sections

Rate five prompt-linked competencies

Interviewers should note the candidate's stated evidence before assigning a rating. Score each dimension independently; do not total or convert the results to an automatic recommendation.

Worked calibration

Oracle example: Saying that the 202 response proves acceptance leaves a material gap because the supplied operation is still pending and the product contract requires a durable membership. A candidate who distinguishes those states and proposes checking the operation plus persisted membership provides workable evidence; do not require one assertion library or database.

NO example: If the session ends before the diagnosis prompt, record NO for that competency with “prompt not reached.” Do not infer it from the candidate's earlier test isolation answer.

Scale: 1 = material gap remains after the fixed prompt; 2 = partial evidence with a significant gap; 3 = workable evidence for the stated role scope; 4 = level 3 plus independent handling of an important interaction; NO = not enough fair evidence. Record why NO was used. Do not convert it to zero or infer it from another answer.

Rate each competency only from its matching timed prompt. Equivalent tools and test layers are valid when they observe the stated risk.
Competency and prompt1 · Material gap3 · Workable evidence4 · Strong evidence
Risk prioritization, 5–12Focuses on pass count or low-impact polish while missing duplicate membership or false accepted state.Identifies at least one user-impacting risk, orders a next check and names an uncertainty or owner.Separates duplicate effects from misleading feedback, prioritizes using user impact and reversibility, and requests the smallest decisive next evidence.
Oracle quality, 12–20Treats any 2xx, toast or mock as proof of accepted membership.Separates operation acknowledgment from durable membership and UI state; proposes an assertion at an appropriate service boundary.Connects idempotent request identity, operation completion, persisted membership/seat invariant and truthful pending/accepted UI, without requiring one test framework.
Test isolation, 20–28Accepts shared mutable fixtures or disables parallelism without identifying ownership and cleanup risks.Gives each run unique workspace/invitation data or an equivalent isolated resource, with bounded cleanup.Explains worker/test lifecycle, safe cleanup after failure, collision diagnostics and where deterministic parallelism is worth keeping.
Failure diagnosis, 28–38Retries until green, adds a fixed sleep, or labels every failure a flake without preserving evidence.Correlates request/operation IDs, status, trace and fixture lifecycle; retains the first failure and proposes a reproducer.Separates contract change, product defect and fixture race using discriminating checks; assigns each finding to its owner without overclaiming causation.
Evidence limits, 38–48Claims the suite proves the release safe or assigns unilateral release authority to QA by default.Names what mocked and failing tests do and do not establish, and identifies the accountable product/engineering release owner.States a bounded release recommendation with unresolved evidence, explicit decision owner and a proportionate follow-up or exception record.

For every competency, 2 means relevant reasoning is present but an important gap remains or substantial prompting was needed. NO means the core prompt was skipped, materially changed, interrupted or unclear, so evidence was not fairly elicited. Record the reason and use the same follow-up opportunity for candidates in this stage.

The synthetic example is not a validated selection instrument or a guarantee of release quality. Have subject-matter reviewers confirm that the competencies match the actual job, dry-run the timing and wording, and use the same adjustment and scoring process for all candidates in the stage.

↑ Back to sections

Print the self-contained interviewer packet

This packet repeats the complete scenario, format and prompts. It adds rating anchors and a blank evidence sheet.

The interviewer artifact can be printed on its own.

Software QA / test automation engineer discussion, fixed 60-minute agenda

Use the same synthetic facts and five timed prompts for each candidate. This is a discussion aid, not a validated test or automatic ranking instrument.

Fixed timing and five scored competencies.
TimeStageCompetency
0–5OrientUnscored
5–12Prompt 1Risk prioritization
12–20Prompt 2Oracle quality
20–28Prompt 3Test isolation
28–38Prompt 4Failure diagnosis
38–48Prompt 5Evidence limits
48–55Candidate questionsUnscored
55–60CloseUnscored

Scenario: invitation acceptance in workspace W-31

A user accepts invitation I-17 while a workspace has one seat left. The product contract says an invitation is “accepted” only after the service confirms exactly one durable membership and reserves that seat. While work is pending, the interface must say so. The candidate is assessing release evidence, not being asked to implement the feature.

All facts below are synthetic and supplied for this exercise. Do not infer facts that are not listed.
EvidenceObservation
API contract noteThe acceptance endpoint changed from HTTP 201 to HTTP 202. A 202 returns an operation URL. The operation can be PENDING before it becomes SUCCEEDED. Requests must carry an Idempotency-Key; repeated requests with the same key resolve to one operation. Requests without a key are not guaranteed to deduplicate.
Unit run42 of 42 tests pass. The acceptance client is stubbed to return HTTP 201 with status: accepted. No service or persisted membership is exercised.
Browser run PW-128 of 8 tests pass. The POST is intercepted and returns HTTP 201. The test clicks once and checks that an “Invitation accepted” toast appears. It does not reload or read durable membership state.
Browser run PW-18Against the updated service, the POST returns HTTP 202 and an operation URL. The operation remains PENDING for 1.1 seconds, then becomes SUCCEEDED. The current interface shows “Invitation accepted” as soon as it receives any 2xx response. The test expecting HTTP 201 fails.
Parallel run PW-21Two workers use the shared workspace fixture W-31 and invitation I-17. Worker B cleanup deletes I-17 while worker A polls its operation. Two of ten first attempts return 404; both pass on retry.
Request trace R-44During one delayed acceptance, the user clicks twice 400 ms apart. The browser sends two POST requests, both without an Idempotency-Key, and the service logs two operation IDs. The trace does not show whether either request created a durable membership.

There is no production or customer data in this exercise. No browser runner, source code, database access or specific testing tool is required. You may ask for a stated fact to be repeated, explain an assumption, or say which owner must supply missing information.

Five fixed timed prompts

  1. 5–12 minutes: Which observations most change your release risk, what would you investigate first, and what fact or owner do you need next?
  2. 12–20 minutes: What would a passing check need to observe for this workflow, and where would you observe it? Which current checks would you keep or change?
  3. 20–28 minutes: How would you make parallel runs reproducible without letting one run's setup or cleanup affect another?
  4. 28–38 minutes: Walk through how you would investigate the first failure from the supplied evidence. What would you capture next, and how would you preserve the first result?
  5. 38–48 minutes: Given these observations, what can the team conclude and what remains unknown? Who should decide whether to release, and what evidence should inform that decision?

Rating anchors

Scale: 1 = material gap remains after the fixed prompt; 2 = partial evidence with a significant gap; 3 = workable evidence for the stated role scope; 4 = level 3 plus independent handling of an important interaction; NO = not enough fair evidence. Record why NO was used. Do not convert it to zero or infer it from another answer.

Rate each competency only from its matching timed prompt. Equivalent tools and test layers are valid when they observe the stated risk.
Competency and prompt1 · Material gap3 · Workable evidence4 · Strong evidence
Risk prioritization, 5–12Focuses on pass count or low-impact polish while missing duplicate membership or false accepted state.Identifies at least one user-impacting risk, orders a next check and names an uncertainty or owner.Separates duplicate effects from misleading feedback, prioritizes using user impact and reversibility, and requests the smallest decisive next evidence.
Oracle quality, 12–20Treats any 2xx, toast or mock as proof of accepted membership.Separates operation acknowledgment from durable membership and UI state; proposes an assertion at an appropriate service boundary.Connects idempotent request identity, operation completion, persisted membership/seat invariant and truthful pending/accepted UI, without requiring one test framework.
Test isolation, 20–28Accepts shared mutable fixtures or disables parallelism without identifying ownership and cleanup risks.Gives each run unique workspace/invitation data or an equivalent isolated resource, with bounded cleanup.Explains worker/test lifecycle, safe cleanup after failure, collision diagnostics and where deterministic parallelism is worth keeping.
Failure diagnosis, 28–38Retries until green, adds a fixed sleep, or labels every failure a flake without preserving evidence.Correlates request/operation IDs, status, trace and fixture lifecycle; retains the first failure and proposes a reproducer.Separates contract change, product defect and fixture race using discriminating checks; assigns each finding to its owner without overclaiming causation.
Evidence limits, 38–48Claims the suite proves the release safe or assigns unilateral release authority to QA by default.Names what mocked and failing tests do and do not establish, and identifies the accountable product/engineering release owner.States a bounded release recommendation with unresolved evidence, explicit decision owner and a proportionate follow-up or exception record.

For every competency, 2 means relevant reasoning is present but an important gap remains or substantial prompting was needed. NO means the core prompt was skipped, materially changed, interrupted or unclear, so evidence was not fairly elicited. Record the reason and use the same follow-up opportunity for candidates in this stage.

Blank evidence sheet

Record specific evidence first. Choose NO with a reason when the prompt did not fairly elicit evidence.
Competency and timeObserved words, choices and assumptionsRatingMissing evidence / consistent follow-up
Risk prioritization, 5–12


1 / 2 / 3 / 4 / NO

Oracle quality, 12–20


1 / 2 / 3 / 4 / NO

Test isolation, 20–28


1 / 2 / 3 / 4 / NO

Failure diagnosis, 28–38


1 / 2 / 3 / 4 / NO

Evidence limits, 38–48


1 / 2 / 3 / 4 / NO

Independent debrief notes

Evidence tied to actual role outcome:

______________________________________________________________

Different readings and relevant anchor:

______________________________________________________________

Prompt missed or fair follow-up needed:

______________________________________________________________

Open product, test-data or release-authority boundary:

______________________________________________________________

Candidate / role: ______________________________ Interviewer: ______________________________

Before using this scorecard

  • Confirm the five competencies match the actual work and level.
  • Name product, engineering and release decision owners before the interview.
  • Dry-run the scenario and timing with synthetic facts; no candidate or predictive validation is claimed.
  • Share the format and reasonable adjustment route in advance.
  • Retain notes under the organization's candidate-data rules.

This scorecard is a discussion aid, not a validated test or automatic candidate ranking. QA provides evidence; the named product or engineering release owner remains accountable for the release decision unless an explicit policy says otherwise.

↑ Back to sections

Choose your next step