How we catch a 1% regression

Felix has one rule about performance: a new release must not be measurably slower than the previous one. Not “no big regressions”. No regression we can measure, in any cell we measure.

That sounds simple until you try to measure 1% on a shared cloud machine. The same commit, benchmarked twice on GitHub Actions, came back with a median latency of 68 us one time and 184 us the next. Against noise like that, a 1% loss is invisible unless you design for it.

How it works

Every pull request runs a workflow called perf-pr.yml. It checks out the PR's merge-base as the baseline and the PR head as the candidate. Then it builds both and benchmarks both, back to back, in the same job on the same runner instance.

That last part matters most. A comparison against a stored number from last week compares two machines as much as two commits. Running both sides on one machine, minutes apart, cancels most of that. Whatever the hardware and its neighbours are doing, both sides see it.

Each side runs a small subset of the full matrix: one-at-a-time publishes of 256 bytes, batched publishes of 1 KiB, fan-out to one and ten subscribers, and a 64 KiB cell that reaches the subscriber byte bound. Each cell runs five trials by default. For a release check we raise that to 15 or 20.

The two checkouts get separate cargo target directories. Cargo identifies a path dependency by name and version, not by which checkout it came from, so a shared target directory can let the candidate link the baseline's build. The job would then benchmark the merge-base twice and report no change.

Both sides are also driven by the PR's own harness and workload files. Otherwise a change to the harness would show up as a change in the broker.

Deciding what counts

A script, scripts/perf/compare_benchmarks.py, compares each metric with Welch's t-test. Welch's version doesn't assume both sides have the same variance, which matters here because a change can make a cell noisier as well as slower. It is implemented in plain Python so the job needs nothing installed.

Every row ends up in one of three states:

is_regression = (
    worse_direction
    and p_value is not None
    and p_value < alpha              # 0.05
    and abs(pct_change) >= min_effect_pct   # 3%
)
is_inconclusive = (
    not is_regression and worse_direction and abs(pct_change) >= inconclusive_pct  # 10%
)

A regression has to be both significant and big enough to care about. The 3% bar is there because with enough trials a t-test will call almost anything significant. The inconclusive class exists for the opposite failure. With five trials and a wide spread, the test cannot tell a doubling from noise, and “not significant” would read as “passed”. So a large change in the wrong direction that misses significance gets a grey question mark, not a green tick. Each row also shows the coefficient of variation for both sides, so you can see when the spread is bigger than the effect.

The check is advisory. It never fails a PR. A shared runner's noise floor isn't good enough to block merges on.

Why a single run lies

Look at what the same code does across runner classes. Before a release, we run the old release against the new one on three GitHub runner types, 20 trials each. For the previous release, one-at-a-time publishes to ten subscribers had a median latency of 84 us on ubuntu-latest, 116 us on ubuntu-22.04 and 219 us on ubuntu-24.04-arm (#1317, #1318, #1319). Throughput went from 12,408 to 4,674 messages a second. That's a factor of 2.6 from the runner alone.

You don't even need a different runner class. In #1286 the same two commits ran twice. The first run reported a p50 of 68 us for a single subscriber. The rerun reported 184 us. Nothing changed but the machine we were handed.

Tail latency is worse. On that first run, p999 for ten subscribers went from 448 us to 821 us, +83%. The rerun, same code, 15 trials, said +4%. The first number was flagged inconclusive, which is exactly what that class is for. A p999 over 15 trials rests on a handful of samples per trial, and one slow scheduler tick moves it.

Batched cells have a different problem. A run that sends batches of 64 from eight publishers is saturating the broker, around 700k messages a second. Its latency is mostly the depth of the queues, and queue depth follows how fast the runner is. On PRs that never touched the broker, that cell's p50 swung from 7.9 ms to 34 ms across runs of main, and one PR with an identical broker binary got flagged as a regression (#1257).

So we read batched latency as a throughput profile, not request latency. Its throughput is a real signal. Its percentiles mostly measure how long the line was.

How we found a 1-2% loss

The A/B of main against the last release came back with throughput for one-at-a-time publishes 1-2% lower, for one subscriber and for ten. That's under the 3% bar, so the comment showed green ticks. But it was significant and pointed the same way on both cells, and a rerun said the same: -2.4% (p=0.003) and -2.1% (p=0.03).

We bisected it in Actions with throwaway draft PRs. Each one sets the old release as its base branch and one commit from main as its head. The workflow does the rest: same runner, back to back, 15 trials. The PR gets a comment with the table, and we close it unmerged.

The commit just before #1243 matched the release (+0.6% and +0.7%). #1243 itself did not (-2.2% and -0.8%). That change had moved where the feeder gives a batch's bytes back to the subscriber's byte bound. It released them, a semaphore release, as soon as it took the batch, so the release sat on the delivery path before the frame was encoded.

We tested ideas the same way before touching the code. A lock-free version of the release, for example, did nothing measurable.

The fix in #1309 splits the subscriber's hold off the batch and drops it at the end of the pass, after the frame is in the lanes. Against the old release that measured -0.4% (p=0.45) and -0.0% (p=0.83), and the 64 KiB cell came out 10.8% faster.

What it costs

Runner time, mostly. One A/B takes 13 to 27 minutes, depending on the runner class and the trial count. A release check is three of them. A bisect is one per commit you test, plus one per hypothesis you rule out.

Then there's waiting. Every step of a bisect is a PR that has to queue, build twice and run. You can't do it in an afternoon at your desk.

The bigger cost is discipline. A 1.5% dip under a 3% bar shows a green tick. It's easy to call it noise and ship. Doing the bisect means treating a number the tool calls fine as a problem until it's explained.

Here are the results from the last release check, the old release against the new one, 20 trials on each runner class:

The last three columns are throughput change. The 1 KiB column never crosses the 3% bar, but it's significant on every runner class. So we bisected it the same way, and this time the results didn't line up with any commit. Two neighbouring commits came out +1.6% and -1.4%. Main itself gave -3.3% in one job and +1.6% in another.

Then came the control that explained it. We took the old release and added one black_box call to a cold startup function, a change that does nothing at runtime. It measured -2.0% (p=0.009) in one job and +2.0% (p<0.0001) in another. Felix builds with fat LTO and a single codegen unit, so any change re-lays-out the whole binary, and code layout alone moves this cell by about 2% either way. A single job's p-value measures the noise between runs of one build. It knows nothing about the build.

What we learned

Repeating across runners rules out the runner, not the build. The 1 KiB dip was significant on three runner classes whose speeds differ by 2-3x, and it was still noise. Every job compiled the same deterministic binary, so every job inherited the same code layout. Only rebuilding with a change that does nothing showed how much layout alone moves a cell.

Find the noise source before arguing about the result. The batched latency rows looked like regressions on PRs that never touched the broker. They were queue depth on a variable machine. The p999 swing was a few samples. Each time, the useful work was finding out why the number moved, not debating whether it was real.

A threshold is a filter for the bot, not a rule for the person reading it. The 3% bar keeps the PR comment from crying wolf. The release rule is stricter than that, and the gap between them is filled by reruns and bisects.

And bisecting in CI works better than it sounds. A throwaway PR is cheap to make, gives the same runner to both sides, and leaves its result in a comment anyone can read later.

What we'll do differently

Every release check runs on all three runner classes at 20 trials, not one class at 15. Each side is built at least three times, with a harmless change that shifts the layout, and the comparison uses the median of each build. That puts build-to-build variance inside the test instead of outside it.

We'll run A/A checks, the old release against itself, so we know the noise floor of each cell before we read an A/B.

And any dip that is significant on more than one runner class gets bisected before the release, whatever the bar says.

Trying it

docker compose up -d from the repo root starts a control plane and a broker from the published images (Docker Compose guide). The benchmark harness, the subsets and the comparison script are in scripts/perf/, the workflow is .github/workflows/perf-pr.yml, and the benchmarks page explains the profiles.

docker compose up -d