Your Test Suite Has Three Time Budgets: 2s, 60s, 10 Minutes

I'll research this before writing.Your test suite takes four minutes. So you stopped running it locally about three weeks ago, and now you find out things are broken from CI, or from a user, or from Claude Code confidently telling you the refactor is complete. This is not a discipline problem. It's a latency problem, and it has a number attached to it.

The number isn't one number. It's three, and they're set by what the test run is interrupting — not by how big your codebase is or how virtuous you feel.

The three budgets

Inner loop: 2 seconds. This is the run that happens on save, or the one your agent fires after every edit. Jakob Nielsen's response-time work — still the standard reference, and based on human factors research that predates the web — puts one second as the limit for a user's flow of thought staying uninterrupted, and ten seconds as the limit for keeping attention on the task at all. Test runs aren't UI, but the mechanism is identical: past ten seconds you tab away, and once you tab away the feedback loop is dead. Two seconds gives you headroom.

Pre-push: 60 seconds. Long enough to read the diff you're about to push. Bazel encodes something similar — Google's test-size taxonomy has always been about time, with small tests running "on the order of seconds" and medium tests on the order of minutes.

CI on a PR: 10 minutes. This is the XP ten-minute build, and Fowler still recommends it in the CI article. Past ten minutes people start opening the next PR while the first one is still running, and now you've got two changes in flight and no idea which one broke the build.

If a run exceeds its budget, you will stop trusting it. Not consciously. You'll just start typing --no-verify and it'll feel reasonable every single time.

The AI part makes this worse, fast

Two things changed when we started building with agents.

First, suites got big quickly. Ask Claude Code to add tests and it will add tests — thorough, well-named, and often four of them where one would do. Nobody's pruning. A codebase that would have accumulated 80 tests in six months now has 400 in three weeks, and nobody has ever looked at the total runtime.

Second, and worse: a slow suite teaches your agent bad habits. When the full run is slow, you write CLAUDE.md instructions like "run the specific test file, not the whole suite" — sensible — and then the agent makes a change that breaks something two modules over and reports success. Or the run hits a tool timeout, the agent sees a truncated failure, and starts guessing. The failure mode I see most often in our Discord: the agent decides the test is wrong and edits the assertion. It's not being lazy. It got an ambiguous signal and picked the interpretation that let it finish.

You cannot fix that with a sternly worded prompt. You fix it by making the honest path cheap.

A real one: 3m50s of Postgres containers

Here's a suite I actually untangled. Node API, Vitest, Postgres, ~340 tests, 3 minutes 50 seconds on my machine. Absolute numbers will differ on yours; the shape won't.

Step one is always measurement, and almost nobody does it before they start "optimising".

# Vitest: flag anything over 300ms and print per-test timings
npx vitest run --slowTestThreshold=300 --reporter=verbose

# pytest equivalent: the 25 slowest tests, including setup/teardown
pytest --durations=25 -q

The output was brutally clear. 22 test files, each one calling testcontainers to start its own Postgres. Container startup was ~7 seconds. 22 × 7 = 154 seconds — two thirds of the entire run — before a single assertion executed. The remaining 76 seconds was mostly TRUNCATE ... CASCADE across eleven tables between every test.

The actual test logic? Under 20 seconds of it.

This is the normal answer, by the way. It's almost never "our tests are slow". It's one setup cost multiplied by a number you weren't thinking about. Before you shard across CI runners or buy a bigger machine, find the multiplication.

The fix that made it worse

The obvious move is parallelism. Vitest already runs files in parallel, so I set maxWorkers up, pointed every worker at one shared container, and got the suite to 41 seconds.

Then it started failing. Not always — maybe one run in eight, and never the same test twice. Two workers truncating the same tables while another worker was mid-insert.

Forty-one seconds and flaky is worse than three-fifty and reliable. A slow suite gets ignored eventually. A flaky suite gets ignored immediately, because the first thing anyone learns is that re-running fixes it. That's the moment the suite stops being a signal and becomes a tax. Google made this point about end-to-end tests a decade ago and it's aged well: when a test fails and nobody can tell whether it's the code or the test, people stop paying attention.

If you take one thing from this post: never buy speed with flakiness. Isolation first, then speed.

One container, one database per worker

The version that worked: start a single Postgres container in a global setup file, run migrations once into a template database, then give each worker its own database cloned from that template. Postgres does the clone with CREATE DATABASE ... TEMPLATE, which is a file copy and takes milliseconds.

// vitest.globalSetup.ts
import { PostgreSqlContainer } from '@testcontainers/postgresql'
import { execSync } from 'node:child_process'

export default async function setup() {
  const pg = await new PostgreSqlContainer('postgres:16-alpine').start()
  process.env.PG_ADMIN_URL = pg.getConnectionUri()

  // migrate once, into a template nobody writes to
  execSync('npm run migrate', { env: { ...process.env, DATABASE_URL: pg.getConnectionUri() } })
  execSync(`psql "${pg.getConnectionUri()}" -c "ALTER DATABASE ${pg.getDatabase()} IS_TEMPLATE true"`)

  return async () => { await pg.stop() }
}

// setupFiles: runs once per worker
import { beforeAll } from 'vitest'

beforeAll(async () => {
  const id = process.env.VITEST_POOL_ID ?? '0'
  const admin = await connect(process.env.PG_ADMIN_URL!)
  await admin.query(`CREATE DATABASE test_w${id} TEMPLATE app_test`)
  process.env.DATABASE_URL = urlWithDb(process.env.PG_ADMIN_URL!, `test_w${id}`)
})

Then, per test, wrap in a transaction and roll back instead of truncating:

beforeEach(async () => { await db.query('BEGIN') })
afterEach(async () => { await db.query('ROLLBACK') })

Result: 28 seconds, and it has not flaked since. The pytest version of this is the same idea — a session-scoped container fixture, pytest-xdist for workers, and worker_id from the xdist fixture to pick the database name.

Twenty-eight seconds still blows the inner-loop budget. That's the next problem.

Tier by speed, not by philosophy

Stop arguing about what counts as a unit test. Nobody has ever won that argument and the categories don't map to anything you care about. Tier by runtime budget instead, and enforce it.

{
  "scripts": {
    "test": "vitest run --project=fast",
    "test:all": "vitest run",
    "test:watch": "vitest --project=fast --changed"
  }
}

The fast project is everything with no I/O: pure functions, reducers, validators, pricing logic, date maths. In this codebase that was 240 of the 340 tests and it ran in 1.4 seconds. That's the run that goes in the agent loop and on save.

The other 100 tests hit the database and run on pre-push and in CI. And here's the enforcement bit that matters — put a hard timeout on the fast project so it can't rot:

// vitest.config.ts
export default defineConfig({
  test: {
    projects: [
      { test: { name: 'fast', include: ['src/**/*.test.ts'], testTimeout: 500 } },
      { test: { name: 'db', include: ['src/**/*.db-test.ts'], testTimeout: 10_000 } },
    ],
  },
})

A 500ms per-test timeout in the fast project means the first person — or agent — who imports the database client into a fast test gets a red build immediately, with an obvious cause. Without that guard, your two-second suite is a forty-second suite in a month and you won't be able to point at the commit that did it.

Delete tests

The heretical one. Some of your tests are not worth their runtime, and keeping them is an active cost you pay dozens of times a day.

Do the arithmetic on the worst offender. A test that takes 12 seconds, in a suite you run 30 times a day, is six minutes a day, every day. If you can't remember it ever catching a real bug, it's not free — it's the reason you're reading this post.

The candidates, in order:

The snapshot tests your agent generated in bulk. A 400-line snapshot of a React tree doesn't tell you anything failed correctly; it tells you something changed, which you already knew, because you changed it. When it goes red you press u. That's not a test, that's a diff with extra steps.

The duplicate coverage. Ask Claude Code for tests on a validator and you'll get eight, six of which exercise the same branch with different string literals. Keep the boundary cases, delete the middle.

The end-to-end test that duplicates an integration test, which duplicates a unit test. Pick the cheapest level that would actually catch the bug you're worried about.

Do this by inspection, not by coverage tool. Coverage percentage will punish you for deleting redundant tests, which is exactly backwards.

Wire the fast lane into the agent

Now make it the default. In CLAUDE.md:

## Tests
- After every code change, run `npm test` (fast project, ~1.5s). Always.
- Run `npm run test:all` before you tell me a task is done, or if you
  touched anything under src/db/ or src/api/.
- If a test fails, fix the code. Do not edit assertions or add skips
  without asking me first.

That last line does real work. It converts an ambiguous judgement call into an explicit permission request, and you'd be surprised how often that's the difference.

You can also enforce it with a hook rather than trusting the instruction. Claude Code supports PostToolUse hooks in .claude/settings.json that fire after a file edit — the config schema has changed a few times, so check the current hooks docs before pasting, but the shape is a matcher plus a shell command:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [{ "type": "command", "command": "npm test --silent" }]
      }
    ]
  }
}

This only works because the run is 1.4 seconds. Hook a four-minute suite to every file edit and you've built a machine for burning tokens and patience.

What you're giving up

Be honest about the trade. The fast project doesn't touch the database, so a whole class of bug — bad SQL, a missing index causing a timeout, a migration you forgot to write — will not be caught at your desk. You'll find it 28 seconds later on pre-push, or ten minutes later in CI. You are deliberately choosing to catch integration bugs slightly later in exchange for catching logic bugs constantly. For most of what we build, that's the right way round.

The transaction-rollback trick has a sharper limitation: you can't test code that manages its own transactions, and anything relying on COMMIT semantics — advisory locks, LISTEN/NOTIFY, isolation-level behaviour — needs a real committed database. Those tests exist. Put them in the slow tier, keep them few, and accept the per-test truncate cost for that handful.

And the obvious objection: "my agent doesn't get bored, so why do I care?" Because you're the reviewer. If the verify step takes four minutes, you leave, and you come back to a completed task you didn't watch. The agent's patience isn't the bottleneck — your attention is, and it obeys the same ten-second rule it always did. There's also a plain token cost: a verbose failing suite dumps thousands of lines of stack trace into context, and long noisy logs make the next inference worse, not better.

This week

Run the timing command on your main suite. Just that — pytest --durations=25 or vitest run --slowTestThreshold=300. Look at the top five, and find the setup cost that's being multiplied by a file count. It'll be there.

Then carve out a fast project with a hard per-test timeout, even if it only has thirty tests in it to start. Wire it to your agent. The full clean-up can wait; a 1.5-second run you actually trust changes how you work on day one.

If you've got a suite you've given up on, come argue about it with us — discord.gg/3scUHe7B. Someone in there has already fought your exact database fixture.