Stop letting your AI agent optimise the wrong 1.8%

I'll research the current tooling landscape before writing.Search quota is spent, so let me build a real, reproducible example and get actual numbers rather than inventing any.Ask Claude Code to "make this endpoint faster" and it will happily do it. It'll rewrite your serialisation loop, memoise a helper, swap a map for a for, and hand you a confident diff — and your p99 will not move a millimetre. The model isn't lying to you. It just can't see where the time actually goes, so it optimises the thing that looks slow in the source: the nested loop, the repeated string concat, the function called inside another function.

Time in real systems doesn't live where it looks like it lives. It lives in one query, one waterfall, one cold start, one SELECT * that nobody has read in eighteen months. And the only way to find it is to measure — which sounds obvious right up until you measure the wrong thing and spend a week making your fastest code path 8% faster.

The setup: a boring orders endpoint

Here's a case I built to work all the way through. Everything below is real output from real runs, on a single-core Linux container, Python 3.11.15 with SQLite 3.40.1, against a 25 MB table of 200,000 orders. The absolute milliseconds are meaningless to you — different language, different machine, different data. The ratios are the point, and the ratios show up identically in Node, Go, Rails and everything else.

The endpoint: list a tenant's 50 most recent paid orders, formatted for the UI.

SQL = """SELECT id, tenant_id, status, total_cents, created_at
         FROM orders
         WHERE tenant_id = ? AND status = 'paid'
         ORDER BY created_at DESC LIMIT 50"""

def serialise(rs):
    out = []
    for r in rs:
        out.append({
            "id": r[0],
            "tenantId": r[1],
            "status": r[2],
            "total": "$" + str(r[3] // 100) + "." + str(r[3] % 100).zfill(2),
            "createdAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(r[4])),
        })
    return json.dumps(out)

def handler(c, tenant):
    rs = c.execute(SQL, (tenant,)).fetchall()
    return serialise(rs)

Three hundred requests, random tenants:

full request   mean=15.77ms  p50=15.61  p95=16.81  p99=19.34  max=24.40

Now: paste that code into any AI coding tool and ask it to optimise. You know exactly what you'll get back. The serialise function is visibly wasteful — string concatenation in a loop, zfill, an strftime call per row, an append-to-list instead of a comprehension. It's the only thing in the file that looks like work. So that's what gets rewritten.

The profile says something else entirely

Before touching anything, 100 calls under cProfile:

   ncalls  tottime  cumtime  filename:lineno(function)
      100    0.001    1.686  bench.py:47(handler)
      100    1.646    1.646  {method 'execute' of 'sqlite3.Connection' objects}
      100    0.011    0.030  bench.py:35(serialise)
      100    0.009    0.009  json/encoder.py:205(iterencode)
     5000    0.004    0.004  {built-in method time.strftime}

1.686 seconds total. 1.646 of it in execute. The entire serialisation layer — the loop, the strftime calls, the JSON encoding — is 0.030 seconds. 1.8% of the request.

Amdahl's law does the rest of the arguing for you. If serialisation is 1.8% of your time, making it infinitely fast buys you 1.8%. Make it twice as fast and you've won 0.9%, which is inside the noise on the p95.

I built the "obvious" fix anyway, because you should always test the thing you're about to dismiss:

def serialise_fast(rs):
    return json.dumps([
        {"id": r[0], "tenantId": r[1], "status": r[2],
         "total": f"${r[3]//100}.{r[3]%100:02d}",
         "createdAt": r[4]}
        for r in rs
    ])

List comprehension, f-strings, dropped the strftime entirely. Result:

FIX A: faster serialisation    mean=15.84ms  p50=15.74  p95=17.01  p99=18.38

Mean went up by 0.07ms. That's noise, but it's a good kind of noise: it's the sound of a change that did nothing. A real diff, reviewed and merged, that made the response format worse (raw epoch timestamps now) in exchange for zero.

Ask the database what it's doing

One line of actual measurement, aimed at the 97.6%:

EXPLAIN QUERY PLAN
SELECT id, tenant_id, status, total_cents, created_at
FROM orders WHERE tenant_id = ? AND status = 'paid'
ORDER BY created_at DESC LIMIT 50;

SCAN orders
USE TEMP B-TREE FOR ORDER BY

There it is. Full table scan across 200,000 rows, then a temporary B-tree built to sort them, to return fifty. Every request. For every tenant.

CREATE INDEX idx_orders_tenant_status_created
  ON orders (tenant_id, status, created_at DESC);

FIX B: index, original slow serialisation
full request   mean=0.31ms  p50=0.30  p95=0.38  p99=0.51  max=0.66

SEARCH orders USING INDEX idx_orders_tenant_status_created (tenant_id=? AND status=?)

15.77ms to 0.31ms. Roughly 50x, from one line, while keeping the "bad" serialisation code the agent wanted to rewrite. Applying both fixes together gets you to 0.24ms — so yes, the serialisation rewrite is worth something now, about 22% of a request that's already 50x faster. That's the correct order of operations, and it's the opposite of the order you get by reading the code and trusting your gut.

Postgres people: same move, better tools. EXPLAIN (ANALYZE, BUFFERS) gives you the plan the planner actually chose plus the I/O it cost, and Datadog's write-up notes that a high shared-block count is a direct signal your working set has outgrown RAM. pganalyze can collect those plans automatically via auto_explain for every slow query, and their framing matches what I keep seeing in practice — most slow queries come down to a missing index, a bad join order, or the planner working from wrong row estimates. None of those three are visible from the application code. All three are visible in the plan.

Wrong thing #1: ranking by mean instead of total

Here's where "measure" stops being enough and "measure the right thing" starts earning its keep.

Same database, three queries, running at realistic relative frequencies — the list endpoint fires constantly, the tenant total runs on a dashboard, the notes search is a rarely-used admin feature:

query           calls   mean_ms   p99_ms   total_ms  %_of_total
listOrders       4000     0.114    0.220      457.6       93.9%
tenantTotal        30     0.693    0.874       20.8        4.3%
searchNotes        12     0.737    0.920        8.8        1.8%

ranked by mean:   ['searchNotes', 'tenantTotal', 'listOrders']
ranked by total:  ['listOrders', 'tenantTotal', 'searchNotes']

The two rankings are exact reverses of each other.

searchNotes is the slowest query in the system by a factor of six and it is worth almost nothing. It burns 1.8% of database time. You could delete it and save 8.8ms. Meanwhile listOrders — the "fast" one, the one nobody flags — is 93.9% of all time spent in the database, because it runs 4,000 times.

This is why every serious database tool sorts by total, not mean. The standard pg_stat_statements recipe orders by total_exec_time descending, with mean_exec_time alongside it as context — and the useful version adds each query's share as a percentage of the whole workload. Total time tells you where the budget goes. Mean time tells you which single call feels bad. You need both, but you start with total, because a 10% win on 93.9% of your time beats a 90% win on 1.8% by a factor of five.

If you're on Postgres and you don't have this on yet, that's the whole intervention:

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

SELECT calls,
       round(total_exec_time::numeric, 2) AS total_ms,
       round(mean_exec_time::numeric, 2)  AS mean_ms,
       round((100 * total_exec_time / sum(total_exec_time) OVER ())::numeric, 2) AS pct,
       query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

Wrong thing #2: averages hide the users who are angry

Second failure mode, and this one is worse because the dashboard stays green.

I put a cache in front of the slow scan query — a plain dict, 95% hit rate, which is roughly what a warm read cache looks like in production. Three thousand requests:

mean=0.033ms  p50=0.000  p90=0.001  p95=0.001  p99=0.737  max=1.017
requests over 1ms: 1 / 3000

The mean is 0.033ms and it is a lie. The median is effectively zero — a dict lookup, sub-microsecond. The p99 is 0.737ms, more than 700 times the median and 22 times the mean. There is no request anywhere in that distribution that takes 0.033ms. The average describes a user who does not exist.

Scale those numbers up to a real service and this is the "it's fast for me" bug. Your average is 40ms, your p99 is 2.4 seconds, and the 1-in-100 requests eating those 2.4 seconds are — reliably, infuriatingly — your biggest tenants, because they have the most data and the coldest cache entries. The people paying you the most money get the worst experience, and the mean smooths it into invisibility.

Google made this exact call in the web performance world. INP replaced FID as a Core Web Vital in March 2024 because it measures every interaction across a page visit rather than just the first, and captures input delay, processing time and presentation delay instead of input delay alone. And the threshold isn't an average: a good INP is 200ms or less at the 75th percentile, with 200–500ms needing improvement and above 500ms considered poor. Percentile, not mean, on a metric defined by the worst part of the interaction.

Set your alerts on p95 and p99. Keep the mean on the dashboard if you like looking at it. Never make a decision with it.

Wrong thing #3: measuring somewhere the user isn't

Lighthouse on your laptop is a lab measurement: one machine, one network, one run, usually a desktop CPU with a warm cache and no ad scripts. It's a great debugging tool and a bad description of reality.

Real User Monitoring — field data — captures what a site's actual users experience, and it's field data that Google uses to decide whether you meet Core Web Vitals thresholds. The gap between the two is where a lot of wasted optimisation lives: you shave 300ms off a lab LCP by inlining critical CSS, ship it, and the field p75 doesn't move because your real bottleneck was a third-party tag on a 4G connection.

Collecting the field version is genuinely small. The web-vitals library is about 2KB and modular, and it exists because the browser's built-in performance APIs don't expose these metrics directly — the library builds them on top. It's designed to match how Chrome measures and reports these metrics to Google's own tools, which matters, because a homegrown timer that disagrees with CrUX is worse than no timer.

import { onLCP, onINP, onCLS } from 'web-vitals';

const send = (metric) => {
  navigator.sendBeacon('/rum', JSON.stringify({
    name: metric.name,
    value: metric.value,
    rating: metric.rating,
    path: location.pathname,
  }));
};

onLCP(send);
onINP(send);
onCLS(send);

Ten lines. Then you have a distribution instead of an anecdote, and you can compute your own p75 per route rather than arguing about which page Lighthouse happened to test.

Making the agent measure instead of guess

The practical fix is to stop asking for optimisations and start asking for evidence. These two prompts produce completely different work:

❌ "This endpoint is slow, can you optimise it?"

✅ "Don't change any code yet. Write a script that runs this endpoint 300 times against seeded data and prints mean/p50/p95/p99. Then run it under a profiler and show me the top 10 functions by cumulative time. Then run EXPLAIN on every query it executes. Report back with the numbers and tell me which single change has the largest expected impact. I'll approve before you edit anything."

The second one takes maybe four minutes longer and it is the difference between the 1.8% fix and the 50x fix. Agents are excellent at building measurement harnesses — it's mechanical, well-specified work with a clear right answer. They're poor at guessing where time goes, because so are we; that's why profilers exist.

Then commit the harness. bench.py, bench.ts, whatever — check it into the repo next to the tests. Now every future "make it faster" request starts with run the benchmark, record the baseline, and every change gets a before-and-after instead of a vibe. Put it in your CLAUDE.md or .cursorrules as a standing instruction: no performance change lands without a measured baseline in the PR description.

The tradeoffs I'm accepting, and the objection you're about to raise

Two honest costs.

Measurement distorts what it measures. That cProfile run reported 1.686 seconds for 100 calls — about 16.9ms each — against an unprofiled mean of 15.77ms. Roughly 7% of overhead, injected by the profiler itself, and it isn't distributed evenly: function-call-heavy code gets penalised more than a single long execute. So a profiler is a tool for finding proportions, not for quoting latency numbers. Use it to answer "where does the time go", then measure the actual improvement with the profiler off. Same discipline applies to EXPLAIN ANALYZE — it's real execution with instrumentation attached, so treat the plan shape as gospel and the timings as approximate.

Percentiles need volume. A p99 over 100 requests is one data point wearing a suit. My 3,000-request run put exactly one sample above 1ms; if I'd run 100 requests I might have caught zero and concluded the tail was fine. Below a few thousand samples, look at the max and the full distribution rather than pretending the p99 is stable.

And the objection: isn't this premature optimisation? No — it's the opposite. Knuth's line about premature optimisation is an argument against optimising without evidence, and it's usually quoted by people skipping the sentence right after it, which is about the value of measurement in identifying the critical part. Profiling first is precisely the discipline he was arguing for. What's premature is the diff that rewrites a serialisation loop because it looked slow.

The genuinely reasonable version of the objection is: most of my endpoints don't need this. Correct. Don't profile everything. Profile the thing that's actually slow, in the order given by total time spent, and stop when it's fast enough. "Fast enough" is a number you should write down before you start.

Do this one thing this week

Pick the single endpoint your users complain about most. Before you change one character of it, get three numbers: its p50, its p99, and its share of total backend time. If you're on Postgres, pg_stat_statements gives you the third one in a single query. If you're on the frontend, drop in web-vitals and let it run for 48 hours.

Then look at the gap between your p50 and your p99. If it's more than about 5x, you don't have a slow endpoint — you have a fast endpoint with a tail, and the fix is somewhere completely different from where you were about to look.

If you find something weird in your own numbers, bring the profile output and we'll dig into it — there's a steady stream of "why is this 40x slower than it should be" threads in the club and they're consistently the most useful conversations we have. discord.gg/3scUHe7B.