Playwright flaky tests: a retried pass still exits 0, so make CI count it

Playwright flaky tests do not fail your build by default. Playwright's release notes say the test runner "exits with code 0 when all failed tests recovered upon a retry", so a test that fails once and passes on retry looks green in CI. This guide, checked against the Playwright 1.64 docs on October 25, 2026, shows how to count those tests, trace them and reproduce them.

Playwright flaky tests are green by default

Playwright Test does not retry anything unless you ask: by default failing tests are not retried. Once you set a retry count, the runner sorts every test into one of three groups. "Passed" means it passed on the first run. "Flaky" means it failed on the first run but passed when retried. "Failed" means it failed on the first run and on every retry.

The catch is what happens to the process exit code. The 1.45 release notes add a flag for it and describe the default: "by default, the test runner exits with code 0 when all failed tests recovered upon a retry." A CI job that only checks the exit code will therefore report success while the output says "1 flaky".

The configuration example on the configuration page shows how easy this is to hit: retries set to 2 on CI and 0 locally, one worker on CI, and trace set to on-first-retry. Nothing in it fails the run when a test needs its retry.

One more detail from the retries page: when a test fails, Playwright discards the whole worker process and its browser, and the retry starts in a new worker. A retry therefore starts from a fresh browser and a fresh process, which is one way a retry can pass without the underlying problem being fixed. In our view, a "flaky" label is a bug report, not a pass.

Step 1: make CI count flaky tests

Playwright gives you two switches for this, both documented. On the command line, --fail-on-flaky-tests fails the run if any test is flagged as flaky; it defaults to false. In the config file, failOnFlakyTests does the same thing, was added in version 1.52, and its documented example is failOnFlakyTests: !!process.env.CI.

Turning it on across a suite that already has flaky tests will turn your main branch red on the first run. In our view, a safer rollout has three moves:

1. Run the suite once with the flag in a non-blocking job and record which tests come back flaky.
2. Fix or quarantine those tests (steps 4 and 5 below).
3. Switch the flag on in the blocking job so a new flake stops the merge instead of joining the pile.

The number to watch before step 3 is the flaky list itself. The retries page shows the report format: a "1 flaky" summary followed by the file, line and title of each test, which is enough to build a list.

Step 2: keep the trace of the first failure

The common advice is to leave trace on on-first-retry. The best practices page says traces are set to run on CI on the first retry of a failed test, and the config above uses it. The TestOptions reference defines it as "Record and keep a trace only for the first retry of a test." That means the trace you get is the retry, and the first failing run, the one you actually need to explain, was not traced.

If your retry passes, the trace you get shows a healthy run. Four trace modes are documented, and they answer different questions:

The last mode arrived in version 1.59, and the release notes describe it as "great for comparing a passing trace with a failing one from a flaky test." Two of the four modes, retain-on-failure and retain-on-failure-and-retries, record a trace for every run before deciding what to keep. The best practices page also says it does not recommend the plain on mode, which traces every test, because it is "very performance heavy", so in our view you should measure your CI time before switching a large suite over. If you only want the first failure, retain-on-first-failure avoids recording the retries.

Step 3: reproduce the flake before you fix it

A flaky test is only fixed when it stops failing under the conditions that made it fail. The command-line reference lists the flags that let you recreate those conditions.

• Repeat it. The --repeat-each flag runs each test N times. Pair it with --retries=0 so a failure shows up as a failure, for example: npx playwright test checkout --repeat-each=30 --retries=0. The file argument is matched as a regular expression against the full test path.
• Remove parallelism. The --workers flag takes a number, and the docs say to use 1 to run in a single worker. If a flake vanishes at one worker and returns at four, another test or a shared resource is involved. The default is 50% of logical CPU cores, which can differ a lot from the single CI worker in the documented config.
• Change the order. New in 1.64, --shuffle schedules tests in random order, "which helps to find tests that accidentally depend on each other." The seed is printed at the start of the run, and passing it back (npx playwright test --shuffle 271828182) reproduces the same order. Tests that must run together, such as a serial suite, keep their order.
• Re-run only the failures. The --last-failed flag runs only the tests that failed last time, reading from a last-run file in the output directory by default.

A quick way to use these together, in our view: run the repeat command at one worker first. If it never fails, raise the workers to match CI. If it still never fails, add --shuffle and keep the seed of any run that fails.

Step 4: find out whether other tests are the cause

In our view, a test that fails with the rest of the suite and passes alone often points to shared state rather than timing. The best practices page says each test "should be completely isolated from another test and should run independently with its own local storage, session storage, data, cookies etc."

Playwright 1.62 added a setting that helps test that idea. The retryStrategy option defaults to 'immediate', which retries a failed test as soon as a worker is free. Setting it to 'isolated' runs retries at the end, "one by one in a single worker", which the docs say minimizes interference between retried tests and the rest of the suite "at the expense of the total run time."

How to read the result is our reasoning, not the docs'. A test that fails in the main run and passes in the isolated retries points toward interference from other tests or from load, which is a clue to follow with --workers=1 and --shuffle, not proof. And if interference is the cause, isolated retries should pass more often, which can make a flake look healthier, so keep failOnFlakyTests on alongside it.

Serial mode is the other thing to check. The retries page says test.describe.serial groups dependent tests, skips the rest of the group after a failure, and retries all of them together, and it adds: "It is usually better to make your tests isolated, so they can be efficiently run and retried independently."

Step 5: fix by cause, and quarantine with an owner

Two causes show up in Playwright's own guidance. The first is timing: the best practices page says web-first assertions such as toBeVisible() wait until the expected condition is met, so in our view an assertion that checks a value once, immediately after an action, is the first thing to replace. The second is the shared state described above.

If a fix will take a while, quarantine the test instead of leaving it in the retry loop. The annotations docs show how to tag a test and skip it from the command line with --grep-invert @tag. The 1.62 release notes added -G as a shorthand for --grep-invert. For a test you do not want run at all, test.fixme() marks it as failing, and "Playwright will not run this test."

A sensible pattern, in our view: tag flaky tests @flaky, exclude them from the blocking job, run them in a separate non-blocking job, and write down an owner and a date. A quarantined test with no owner is a deleted test that still shows up in the report.

How many retries should Playwright use?

The docs do not give a number. They show 2 retries on CI in the configuration example and 3 in the retries example. In our view, one or two is enough for a suite whose purpose is to tolerate an occasional infrastructure hiccup. More than that mostly hides flaky tests, and with failOnFlakyTests on, extra retries only delay the failure.

Where this fits among test tools

The settings above belong to Playwright Test as documented on October 25, 2026. Options and defaults change between releases, so read the docs for your installed version first. Other runners and hosted platforms name their retry and reporting options differently, so check each tool's own documentation before assuming the same exit-code behavior.

As of October 25, 2026, Upfynd lists 144 products in Testing & Monitoring, including Cypress.io, mabl, Katalon and TestMu AI, and 491 in Developer Tooling. If you maintain a testing tool yourself, you can submit it for a free listing.

Your next step

Add --fail-on-flaky-tests to a non-blocking CI job today and read the list it prints. That list is your backlog: pick the test that flakes most, set trace to retain-on-failure for it, and run it with --repeat-each until it fails on demand.