A bug is local, a misunderstanding is everywhere

I'll research current material on AI coding agent failure modes before writing.Your agent shipped code that passes its tests, reads cleanly, and is completely wrong. You point at the wrong bit, it apologises, patches it, and forty minutes later you're on your fourth patch and the numbers still don't add up.

That's not a bug. That's a misunderstanding, and treating it like a bug is why your afternoon disappeared.

Two failure classes that look identical in the diff

A bug is when the intent was captured correctly and the implementation is wrong. Off-by-one. Missing null check. Awaited the wrong promise. The model of the problem in the agent's head matched yours; the code just didn't execute that model faithfully. You point at the line, it changes the line, the failure is gone. Bugs are local.

A misunderstanding is when the implementation is a faithful expression of the wrong model. The code is internally consistent. The variable names are sensible. The tests pass because the tests were written from the same wrong model. Nothing is broken in the way static analysis or a stack trace understands "broken". Misunderstandings are distributed — they live in every file the agent touched, plus the ones it didn't.

The 2025 Stack Overflow Developer Survey puts a number on how common this feels: the biggest single frustration, cited by 66% of developers, is dealing with "AI solutions that are almost right, but not quite," which leads directly into the second-biggest frustration, that debugging AI-generated code is more time-consuming (45%). Read that phrasing again. "Almost right" is not what bugs feel like. A bug is just wrong — it throws, it returns null, it renders nothing. "Almost right" is the signature of a coherent wrong model.

The seat-counting incident

Here's the shape of it. Illustrative, but if you've built a B2B SaaS you've lived some version.

You ask for per-seat billing. Team owners invite members by email, you sync the seat count to Stripe. Reasonable prompt, one paragraph, no spec. The agent produces this schema:

create table memberships (
  id         uuid primary key,
  org_id     uuid not null,
  user_id    uuid,
  email      text not null,
  status     text not null, -- 'invited' | 'active' | 'removed'
  created_at timestamptz not null default now()
);

Good schema. Then the billing sync:

export async function syncSeats(orgId: string) {
  const { count } = await db
    .from('memberships')
    .select('*', { count: 'exact', head: true })
    .eq('org_id', orgId)
    .neq('status', 'removed');

  await stripe.subscriptionItems.update(itemId, { quantity: count ?? 0 });
}

And the plan limit check:

export async function canInvite(orgId: string) {
  const used = await countMemberships(orgId); // same neq('status','removed')
  const plan = await getPlan(orgId);
  return used < plan.seatLimit;
}

And the header component, which calls the same helper. Three files, one rule, applied consistently: a seat is any membership that hasn't been removed.

That rule is wrong. Your pricing page says you bill for people who have joined. An invite that sits unaccepted for three weeks is a seat you're charging for. You find out when a customer emails asking why they're paying for eleven people when six have logged in.

Nothing here is a bug. Every line does exactly what it says. The agent picked a definition of "seat" — silently, plausibly — and then implemented it beautifully everywhere.

The patch loop, which is the actual failure

Your instinct is to treat it like a bug. You paste the customer email into the terminal:

Bug: we're billing for invited users who haven't accepted. Fix it.

The agent finds syncSeats, changes .neq('status', 'removed') to .eq('status', 'active'), runs the tests, reports success. It has fixed the thing you pointed at.

Now you have a new problem. canInvite still counts invites. So an org on a five-seat plan can have five active members and twelve pending invites, and when those invites are accepted, the org silently lands at seventeen seats with a plan limit of five and no enforcement anywhere. The header still renders "17 of 5 seats used". Stripe says 5.

You patch canInvite. Then the header. Then you discover the CSV export and the admin dashboard and the usage-based overage job. Each patch is correct in isolation. Each one is a separate encoding of a rule that now lives in six places and agreed in none of them.

This is the tax. You spent the afternoon doing bug-shaped repairs on a misunderstanding-shaped problem, which means you played whack-a-mole against a definition instead of changing the definition once.

It's worth noting that METR's randomised controlled trial found something uncomfortable in the same territory: experienced open-source developers took 19% longer to complete real tasks when allowed to use AI tools, while estimating afterwards that AI had made them 20% faster. The study didn't isolate misunderstanding-versus-bug as a cause, and I'm not going to pretend it did. But the gap between felt speed and measured speed matches the patch loop exactly: every individual patch feels fast, and the aggregate is slow.

The tell: does the wrongness cohere?

Here's the diagnostic, and it takes about thirty seconds.

Look at how the wrongness is distributed. If it fails in one place and everything else is fine, it's a bug — patch it and move on. If the same wrong assumption shows up in three unrelated files, written three different ways, all agreeing with each other, you're looking at a misunderstanding and you should stop editing code immediately.

Bugs are inconsistent with their own codebase. Misunderstandings are consistent with everything except reality.

Second tell: can you name the wrong thing as a noun? "It's counting invites as seats" is a noun-level disagreement. "It's returning undefined when the array is empty" is not. Noun-level disagreements are always misunderstandings, because nouns are where your domain lives and the model has to guess your domain from vibes.

Ask it to restate before you let it edit

When you suspect a misunderstanding, do not describe the symptom. Ask the agent what it thinks the world is:

Don't change any code. In plain language, tell me:
1. What is a "seat" in this codebase?
2. Exactly where is that definition encoded, file and line?
3. What happens to the count when an invite is sent, accepted, or expires?

Two outcomes. If the answer is muddled and self-contradictory, you probably have a bug plus some confusion, and normal debugging works. If the answer is crisp, confident, and wrong — "a seat is any membership that hasn't been removed, encoded in countMemberships at lib/orgs.ts:34" — you have a clean misunderstanding, and now you know its exact address.

That second outcome is the good one. A confidently stated wrong model is the easiest thing in the world to correct. A vague one isn't.

Repair: define the noun, then audit, then edit

The repair sequence is definition, audit, edit. In that order. Skipping straight to edit is what got you here.

First, write the definition down as a file in the repo, not as a message in a chat that dies with the session:

# Domain: seats

A **seat** is a `memberships` row with `status = 'active'`.

- Invites (`status = 'invited'`) are NOT seats. Not billed, not counted
  against the plan limit.
- A seat comes into existence when an invite is ACCEPTED, never when it
  is sent.
- The plan limit is enforced at acceptance time, not invitation time.
- Stripe subscription `quantity` MUST equal the count of active
  memberships at all times. If they disagree, Stripe is wrong.

Second — and this is the step people skip — run a read-only audit:

Read docs/domain/seats.md. Then search the entire repo for every place
that counts memberships, enforces a plan limit, or writes a Stripe
quantity. For each one, quote the code and state whether it AGREES or
DISAGREES with the definition. Do not change any code yet.

The output of that is a list. That list is the true size of your misunderstanding, and it is always longer than you expected. In this scenario it's six places, not the one you noticed.

Third, collapse the rule into one function so it can't drift again:

// billing/seats.ts — the ONLY place this rule lives.
// See docs/domain/seats.md
export async function countBillableSeats(orgId: string): Promise<number> {
  const { count } = await db
    .from('memberships')
    .select('*', { count: 'exact', head: true })
    .eq('org_id', orgId)
    .eq('status', 'active');

  return count ?? 0;
}

Then have the agent replace all six call sites with it. Now a future misunderstanding about seats is a one-line change instead of a scavenger hunt.

Your tests didn't save you because your tests agreed with the agent

This deserves its own paragraph because it's the bit people find genuinely annoying.

it('counts invited members as seats', async () => {
  await invite(org, '[email protected]');
  expect(await syncSeats(org)).toBe(1);
});

Green. Committed. Reviewed. Wrong.

Tests written by the same agent, in the same session, from the same prompt, inherit the same model of the domain. They are excellent at catching bugs — genuinely, this is where AI-generated tests earn their keep — and structurally incapable of catching misunderstandings. A test suite is a machine for detecting inconsistency between your intent and your code. If the intent itself is wrong, the machine happily certifies it.

The only tests that catch misunderstandings are ones written from a source the agent didn't author: your pricing page, an invoice, a screenshot of a competitor, a paragraph from the customer contract. So paste the actual pricing copy in and say "write tests that assert this document is true."

The tradeoff, stated plainly

Writing domain definitions costs time up front, and most of the time you didn't need it. Most tasks genuinely are "add a loading state to this button", and a docs/domain/ file for that is ceremony.

The worse cost: a written definition that's wrong propagates faster and more consistently than a vague one. If you write down that a seat is an active membership and it turns out finance counts trialling orgs differently, you've now got six call sites confidently wrong instead of six call sites accidentally wrong. A well-specified misunderstanding is harder to spot precisely because it's coherent.

So here's the threshold we use: write the definition when the noun appears in something you can be held to. An invoice. A contract. A pricing page. A compliance doc. A support macro. If the word has a meaning outside your codebase that someone else controls, define it. Otherwise, don't bother.

"Isn't this just 'write better prompts'?"

Partly, and I'd push back on the framing anyway.

Better prompts are a front-loading strategy. They assume you knew the right answer at the start. The seat example is honest about the more common situation: you didn't know what a seat was either. Not precisely. Not until the bill went out and a customer told you. The agent didn't misunderstand you — it resolved an ambiguity you were carrying, silently, in the direction that looked most plausible from the code it could see. That's not a prompting failure, that's your domain being underspecified in your own head.

Which is why the useful skill isn't prompt quality. It's the diagnosis after the fact — the thirty-second check of whether the wrongness coheres, and choosing the repair strategy that matches. DORA's 2025 report on AI-assisted development, built on more than 100 hours of qualitative data and survey responses from nearly 5,000 technology professionals, lands on a similar point from the org level: the returns come from the surrounding system, not the tool. Same thing at the keyboard. The agent is fine. Your definitions are the system.

You'll also see the old requirements-defect cost multipliers wheeled out here — the IBM Systems Sciences Institute numbers about spec errors costing 100x more downstream. I'd leave those alone. The provenance is murky and even sources that cite them hedge on the exact ratios. The direction is obviously right; the numbers aren't ones I'd put in a slide.

Do this one thing this week

Open the last PR your agent wrote that you had to fix more than twice. Look at the fixes. Ask whether they were three bugs or one misunderstanding wearing three hats.

If it was one misunderstanding, find the noun at the centre of it and write ten lines defining that noun into docs/domain/. Then run the read-only audit prompt above and count how many places disagree. That count is the thing to pay attention to — it's the first honest measurement most people get of how far a single wrong assumption travels through an agent-written codebase.

If you end up with a number, come tell us. We keep a running thread of these in the club and the counts are consistently worse than anyone guesses. discord.gg/3scUHe7B if you want in.