Flaky Tests: Fix, Quarantine, or Delete

Every automated suite eventually reaches the same quiet failure point: a test fails, someone re-runs it, it passes, and the team moves on without ever finding out why. Do that often enough and a red build stops meaning anything, since the first response to any failure becomes rerun it, not investigate it. That is the real damage a flaky test does, not the individual failure, but the erosion of trust in every result the suite produces afterward.

What causes flaky tests?

A flaky test fails intermittently against unchanged code, and the cause is almost always one of four things: a race condition where the test runs before the application state it checks is actually ready, shared state left behind by a previous test, a dependency on an external service or environment that is not fully controlled, or a selector that matches inconsistently as the page renders. None of these are random. They are just harder to reproduce than a straightforward logic bug.

The real cost of flaky tests

A single flaky test costs more than the minutes spent re-running a pipeline. Multiply that re-run across every engineer who hits it in a day, add the time spent arguing in a pull request thread about whether a failure is real, and a handful of unreliable tests can quietly consume hours of engineering time a week that never shows up on any dashboard as a QA cost.

The bigger cost is what a flaky test trains a team to do. Once a test has failed intermittently a few times, engineers stop reading its failures closely, and a CI/CD quality gate that is supposed to block a real regression instead gets treated as background noise. That is how a genuine defect slips through: not because no test caught it, but because the test that would have caught it had already trained everyone to ignore it.

The fix, quarantine, or delete framework

Every flaky test deserves a decision, not an indefinite retry setting. The framework has three outcomes, and the right one depends on how well understood the root cause is and how much real signal the test provides.

When to fix it

Fix a flaky test when the root cause is identifiable and the fix is proportionate to the value of the test. A race condition solved by an explicit wait for a real application state, instead of an arbitrary sleep, is almost always worth fixing directly, since the underlying Playwright or Selenium fix is usually a few lines and the test keeps its full value afterward.

When to quarantine it

Quarantine a test when it still provides real signal but the root cause is not yet understood well enough to fix confidently. Move it out of the required pipeline so it stops blocking merges, but track it with an owner and a deadline, since a quarantine list with no deadline attached is just a slower way of deleting a test's usefulness. A quarantined test that has not been revisited in a month is a decision nobody actually made.

When to delete it

Delete a test when it no longer earns its place, either because the behavior it checks is better covered by a different, more stable test, or because the flakiness itself reveals the test was never checking something meaningful in the first place. A test that has been quarantined for months with no clear owner and no plan to fix it is not providing signal, it is providing false confidence that something is being checked when nothing reliable actually is.

A worked triage example

A typical suite audit turns up flaky tests that look similar on the surface but resolve very differently once triaged:

• A checkout test that fails roughly one run in ten: traced to a race condition waiting on a payment confirmation modal. Root cause is clear and the fix is cheap. Fix it with an explicit wait on the modal's real state.
• A search results test that fails only in CI, never locally: traced to shared test data another suite mutates during a parallel run. Root cause is understood but the fix, isolating test data per run, is a bigger change. Quarantine it with an owner and a two-sprint deadline.
• A notification badge count test that fails unpredictably with no clear pattern after a day of investigation: root cause still unclear, and the badge count is already covered indirectly by two other stable tests. Delete it, the coverage it duplicates is not worth the ongoing investigation cost.
• A third-party integration test that fails whenever a sandbox environment is slow: root cause is external and outside the team's control. Quarantine it permanently as a known-external-dependency case, but keep a lighter mocked version in the required suite so the integration point still has coverage.
• A visual regression test that fails on font-rendering differences between CI and local runs: traced to a missing font in the CI image. Fix it, a one-line CI configuration change resolves the whole class of failure.

Four of those five resolve in under a day once actually triaged. The fifth, the permanent external dependency, is the honest exception the framework has to allow for, rather than forcing every flaky test into fix or delete when quarantine with a lighter fallback is genuinely the right long-term call.

Preventing flakiness in new tests

Triage matters less once new tests stop introducing the same failure modes. Three habits catch most of it before a test ever reaches the suite: wait on real application state instead of a fixed sleep, since Playwright's own auto-waiting exists specifically to remove the guesswork sleeps introduce, isolate test data so a test never depends on state another test happened to leave behind, and prefer stable selectors, a data attribute built for testing, over a CSS class or DOM position that changes with an unrelated styling update.

This matters more with self-healing test automation in the mix specifically, since a tool that silently relocates a broken selector can mask the exact selector instability this section is meant to prevent, turning a flakiness signal into a false all-clear instead of a caught defect.

Making this a habit, not a one-time cleanup

A single flaky-test cleanup sprint fixes the backlog that exists today and does nothing about the backlog that starts forming again tomorrow. A recurring review, biweekly is a reasonable cadence for most teams, keeps the quarantine list from becoming a graveyard nobody revisits, and keeps Playwright versus Selenium decisions honest, since a framework does not get blamed for flakiness that was actually a test isolation problem the whole time.

Teams that treat this as ongoing maintenance rather than a cleanup project tend to catch flakiness while the root cause is still fresh in whoever wrote the test's memory, which is exactly when it is cheapest to fix. The same discipline that keeps a suite's audit findings from going stale applies here too, a framework only works if someone actually runs it on a schedule.