Intermittent failures are almost always time or ordering

I'll research current sources on flaky tests and race conditions before writing.The test passed 39 times and failed on the 40th. Nobody touched the code, CI just went red on a rerun of the same commit, and the agent you asked to fix it added a 250ms sleep and called it done.

That's the shape of almost every intermittent failure, and it's why almost every quick fix makes things worse. Flakiness feels random because the trigger isn't in the diff — it's in the schedule. Two things happened in an order you didn't specify, and most of the time the order you got happened to be the one you assumed. The bug isn't "sometimes computers are weird." The bug is that your code has an unstated assumption about time or ordering, and the machine only occasionally disagrees with you.

The research says the same thing, boringly

This isn't a vibe. The most-cited taxonomy of flaky tests, from Luo et al. in 2014, categorised 201 flakiness-fixing commits across ~52 open-source projects, and the top three causes were asynchronous wait (45%), concurrency (20%) and test order dependency (12%) — 77% of cases, all of them time or ordering. A later study of 235 flaky UI tests across 62 projects found async wait was again the largest single category at 45%. A 2026 study of fixed flaky-test issue reports inside SAP HANA found Async Wait spiking by a factor of 2.2 in one quarter, and timeout-related reports dropping sharply after the team moved to a global timeout value.

Different decades, different languages, different scale. Same two families.

So when you hit an intermittent failure, you don't need to think creatively. You need to run a two-question triage:

1. Can the check run before the work finishes? (time)
2. Does the result depend on what else ran, and in what order? (ordering — either between tests, or between two things inside your app)

Everything else is a rounding error. Let's do one all the way through.

A real one: the invite that sometimes didn't take

Node, Fastify, Postgres, a queue-backed worker. Accepting an invite enqueues a job that inserts a workspace membership row. The test:

test('accepting an invite grants workspace access', async () => {
  const invite = await api.post('/invites', { email: '[email protected]' })
  await api.post(`/invites/${invite.token}/accept`, { userId: sam.id })

  const { rows } = await db.query(
    'select * from workspace_members where user_id = $1',
    [sam.id]
  )
  expect(rows).toHaveLength(1)
})

Green on a laptop, forever. Red in CI roughly once every forty runs.

Question one answers itself: the HTTP response returns when the job is enqueued, not when it's done. The test then queries a table that a different process is going to write to, at some point, probably soon. There is no happens-before relationship between the response and the insert. On a warm laptop the worker wins the race every time. On a shared CI runner with four other jobs competing for CPU, it sometimes doesn't.

The tempting fix, and the one an agent will reach for unprompted:

await new Promise(r => setTimeout(r, 250))

This is a bet on a latency distribution you have never measured. Worse, it's a bet you'll lose exactly when it matters — the slow runs are correlated (noisy neighbour, cold connection pool, GC pause), so when one test times out, twelve do. And you've now added 250ms to the wall clock whether or not the worker was ready in 4ms. Do that in sixty tests and you've bought yourself fifteen seconds of CI per run in exchange for a flake you haven't fixed.

Make the boundary observable instead

The fix is to give the test something real to wait on. Best case, the system already knows when the job is done, and you just have to expose it:

const { jobId } = await api.post(`/invites/${invite.token}/accept`, {
  userId: sam.id,
})
await worker.waitForJob(jobId) // resolves on complete, rejects on failure

Now the test waits exactly as long as the work takes, and — this is the part people skip — it fails loudly when the job throws, instead of silently reporting "0 rows" and sending you hunting through the wrong file.

When you can't get a handle, poll a predicate against a deadline:

async function until<T>(
  probe: () => Promise<T | null | undefined>,
  { timeoutMs = 5000, intervalMs = 25, label = 'condition' } = {},
): Promise<T> {
  const deadline = Date.now() + timeoutMs
  let last: unknown
  while (Date.now() < deadline) {
    try {
      const value = await probe()
      if (value != null) return value
    } catch (err) {
      last = err
    }
    await new Promise(r => setTimeout(r, intervalMs))
  }
  throw new Error(
    `timed out after ${timeoutMs}ms waiting for ${label}` +
      (last ? ` (last error: ${last})` : ''),
  )
}

const membership = await until(
  async () => {
    const { rows } = await db.query(
      'select * from workspace_members where user_id = $1',
      [sam.id],
    )
    return rows[0] ?? null
  },
  { label: 'workspace membership row' },
)
expect(membership.role).toBe('member')

Look at what changed. A sleep is slow when the system is fast and too short when the system is slow — both directions wrong. A polled deadline returns in ~25ms on a good day and still passes on a bad one, and when it genuinely fails it tells you what it was waiting for. Five seconds sounds generous because it is: the timeout is not a performance assertion, it's a "this will never happen" guard. If you want to assert latency, write a latency test.

This is exactly the trade the browser tools made for you already. Playwright runs actionability checks before an action and auto-waits for them, failing with a TimeoutError if they don't pass in time, and its assertions retry until they match. The pattern generalises to your API tests, your queue, your cache. Wait for a condition, never for a duration.

The tradeoff I'm accepting: polling means the assertion is eventually-consistent, so a bug where the row appears twice, one second apart, still passes. If that matters, assert the count after the wait, not just existence.

The other fix: delete the asynchrony

Sometimes the right move is to run the worker inline in tests, so enqueue means execute. It's fast and completely deterministic. It also means you're no longer testing serialisation, retry behaviour, or the enqueue path at all.

What we do, and what I'd suggest: run inline for the couple of hundred tests that are really about business logic, and keep three or four tests that exercise the real queue end to end with a generous deadline. You get determinism where you need speed and fidelity where you need truth. What you don't get is the illusion that your 200 fast tests prove the queue works.

Family two: ordering between tests

Now the other half. A test that passes alone and fails in the suite is not flaky in any interesting sense — it's coupled, and you just haven't found the partner yet.

The usual culprits are module-level caches, a mocked clock never restored, a connection pool with a leftover open transaction, or a fixture that inserts a row and doesn't clean up. Then the runner's ordering changes — new file name, parallel sharding, a test added above yours — and the coupling surfaces as "random" failure on a commit that has nothing to do with it.

You don't have to wait for that. Shuffle on purpose. pytest-randomly shuffles at the module level, then class, then function, resets random.seed() before each test, and works with pytest-xdist. Go has -shuffle=on. The important part is that every one of them prints or accepts a seed, so a shuffled failure is still reproducible:

# reproduce the exact order CI hit
pytest --randomly-seed=1653322800

# confirm it's ordering, not time: run the suspect alone
pytest tests/test_billing.py::test_seat_count -p no:randomly

# it passes alone? then bisect the preceding tests
pytest --randomly-seed=1653322800 --collect-only -q > order.txt

Once you have order.txt, halve it, keep the failing half, halve again. Four or five rounds gets you from 900 tests to the guilty pair. Then the fix is usually one line: reset the cache in a fixture, roll back the transaction, restore the clock.

This is genuinely good work to hand to Claude Code, by the way — "here's the ordered test list and the failing seed, bisect it and tell me the minimal pair that reproduces" is a mechanical loop with an unambiguous success signal. That's the kind of task agents are actually great at.

Family three: ordering inside your app, which is the one that matters

Here's the seat-limit bug that a 1-in-200 flake found for us. The handler read the count, checked the limit, then incremented:

select seats_used, seat_limit from workspaces where id = $1;
-- app decides: if seats_used < seat_limit then allow
update workspaces set seats_used = seats_used + 1 where id = $1;

Two invite acceptances landing in the same few milliseconds both read seats_used = 9 against a limit of 10, both decide there's room, both increment. You've sold eleven seats. The test only caught it because one test happened to fire two accepts concurrently, and only when the two queries interleaved just so.

The fix is to stop making the decision in application code and let the database do the ordering:

update workspaces
set seats_used = seats_used + 1
where id = $1 and seats_used < seat_limit
returning seats_used;

Zero rows back means over limit — return 409. One statement, one lock, no window. Then add the backstop so it can never be wrong again even if someone writes a new code path:

alter table workspaces
  add constraint seats_within_limit check (seats_used <= seat_limit);

That flaky test wasn't noise. It was the only place in the system where a real production race condition was visible, and the sleep-and-retry patch would have deleted the evidence. This is the actual cost of treating flakes as an annoyance: you are systematically discarding your concurrency bug reports.

Why AI agents get this wrong by default

Ask an agent to fix a flaky test and it will add a retry, bump a timeout, or insert a wait. Not because it's dumb — because that's the overwhelmingly common pattern in the training data, and because it satisfies the objective you gave it. Red became green. Task complete.

You have to make the objective harder. This is the prompt we use:

This test fails roughly 1 in 40 runs in CI and always passes locally.

Constraints: do not add sleeps, do not add retries, do not increase
any timeout. Those are all off the table.

First, classify it and justify the classification:
(a) time — an assertion can run before the work it checks has finished
(b) ordering between tests — the result depends on what ran before
(c) ordering inside the app — two operations can interleave
(d) environment — the infrastructure itself is unreliable

Then write out the exact interleaving that produces the failure, as a
numbered sequence with both actors named, e.g.
  1. request handler enqueues job 7
  2. test queries workspace_members -> 0 rows
  3. worker inserts membership row
If you cannot produce that sequence from the code, say so and tell me
what you'd need to instrument. Do not guess.

The forcing function is step two. An agent that can write the interleaving has understood the bug and its patch will be right. An agent that can't will now tell you it can't, instead of shipping a confident sleep. That single constraint — "show me the interleaving" — has changed more outcomes for us than any model upgrade.

"But sometimes it really is the environment"

Yes. DNS fails, a registry pull times out, a runner's disk fills, a hosted service has a bad five minutes. Category (d) is real and you shouldn't gaslight yourself out of it.

It has a different signature, though, and the signature is easy to check. Environmental failures hit unrelated tests in the same run. They cluster in time across branches. The error surfaces at the transport layer, not the assertion layer — connection reset, not "expected 1, received 0". And they don't reproduce under repetition on a healthy runner. Genuine time and ordering bugs do the opposite: same test, uncorrelated with everything else, fails on the assertion line, and reproduces if you run it 200 times or shuffle the order.

So run the check before you blame the cloud. pytest --count=200 on the single test, or the suite shuffled five times with printed seeds. If it never reproduces in 1,000 local runs but fails weekly in CI, believe the infrastructure story. If it reproduces on attempt 63, you have a scheduling assumption to find.

And if you must quarantine to keep shipping — fine, we've all done it — quarantine with an owner and an expiry date, and keep recording the failures with their seeds. A retry that swallows the failure without logging it doesn't buy you time. It burns your only evidence.

Do this one thing this week

Turn on random test ordering with a printed seed, and run your full suite five times on a throwaway branch. pytest -p randomly (it's on by default once installed), go test -shuffle=on, or the equivalent in your runner. Don't fix anything yet. Just collect which tests fail and on which seeds.

You will find coupling you didn't know existed, and you'll find it while the responsible commit is still recent enough that someone remembers writing it. That list is your backlog, and it's a much better use of an afternoon with an agent than another round of "make the red go away."

If you want to compare notes — or you've got a flake that genuinely doesn't fit either family, because those are the interesting ones — we talk about this kind of thing most days in the club: discord.gg/3scUHe7B. Bring the seed.