Before you loosen the timeout, run the old commit

Felix's cluster tests start real brokers and a real control plane, then publish, kill, freeze and move things. Now and then one fails once in CI and passes on re-run. The reflex is to call it a timing flake and raise the timeout.

In Felix that reflex has been wrong almost every time. Each failure that looked like load on a slow runner pointed at a defect in the broker. I now have one rule: before I loosen a bound, I run the test against a commit from before the change I suspect, and compare.

A publish refused for a leader that never left

The first case was a_quorum_publish_during_a_copy_is_not_held_by_it. It failed in CI with "shard leadership moved before the batch could reach a quorum". Leadership had not moved. The broker log showed the owner opening the shard at a new generation 20 ms before the failure, still as leader.

Here is the part of Felix involved. When a shard is moved to another broker, the destination is staged first, and staging starts a new generation under the same leader. A quorum mark, the record of how far a majority of replicas hold the log, belongs to one generation. The leader has no mark for the new generation until its first replication pass reports in.

The wait for a quorum read "no mark at this generation" as "I am not the leader" and refused at once. Any Quorum publish landing in that gap was rejected, and so was one already waiting when the old mark was replaced.

The fix is MarkTable::wait_while_leading. It follows the generation this broker leads. If there is no mark yet, it waits for one, and it re-checks every 50 ms that the broker still leads the shard. Only losing leadership ends the wait with NotLeading (#796).

The test only failed under heavy CPU starvation. With 16 busy processes on 16 cores, 26 of 30 runs failed that way on main and none did with the fix. With 8 busy processes, 20 of 20 passed either way. A bound that was loose enough for 8 processes would have hidden a real refusal that production could hit on a bad day.

Healthy subscribers dropping events

The second was a demo test that checks that a stalled consumer only loses its own events. Healthy subscribers lost between 0.15% and 1.0% of theirs, against a bound of 0.1%. It looked like a tight bound on a loaded machine.

Felix gives every subscriber a bounded queue, and with the default drop_new policy a full queue drops new events instead of slowing the publisher. That isolation is the point of the design. So a healthy subscriber losing events means something is filling its queue faster than it can read.

This was the A/B habit in full. I ran the test interleaved, A then B then A then B, on one machine with its load running:

The old commit never lost an event, and the knob that restores the old queue depth removed the drops on main. That pointed at the scheduler, and the broker counter felix_sub_queue_dropped_total showed drops on healthy subscribers.

The cause was an executor loop. With a backlog queued, a publish executor takes the next job and fans it out to subscribers without ever suspending. The feeders it had just woken could not run until it stopped, so their queues overflowed even though they read fast enough. The fix is one line, a yield_now() after every job. A single-threaded test queues 32 publishes behind a subscriber queue of 4 and checks that the subscriber receives all 32. With the line commented out it received 4 (#802).

A lease that was checked too late

The third is a test of a frozen leader. The test slows a broker's clock to 1/100, freezes the process with SIGSTOP for about a second, resumes it, and expects the broker still believes it holds its lease. Once, it didn't: the broker logged lease expired 50 ms after resuming.

The guess in the issue was that the freeze outlasted the lease on the wall clock, a slow-runner story. The logs said otherwise. At 1/100 speed, a 1.2 s freeze is 12 ms on the broker's clock, so the lease could only have expired if the broker had not yet read the slowdown when it froze.

The clock-fault file is only re-read when the process reads the clock. The harness waits 250 ms and assumes the lease refresh loop reads the clock about that often. It did not. That loop picked its period once at startup, from a placeholder 10 s lease: 470 ms. The real 1 s lease, adopted on the first heartbeat, never changed it. So an idle leader could go more than 250 ms without a clock read.

Safety was never at risk, because the commit-time check reads the clock directly. But the cached admission flag, the felix_broker_lease_held gauge and the notification that ends a deposed leader's readers could be 470 ms late instead of 47 ms. The refresh loop now recomputes its period from the current lease on every pass (#1088).

One that really was the test

Not every failure is a product bug. A consumer group test polled right after three acknowledged publishes and got only the first record. A poll that returned nothing or part of the data looked like a visibility race.

It reproduced in 229 of 300 runs with six copies in parallel. The broker in that test acknowledges a Leader publish once it is queued, before the write, and docs/semantics.md already said so. Nothing in the group path was wrong. The test now asks for commit acknowledgements, and the docs say plainly that an enqueue ack does not make a record readable (#1038).

I count that one in favour of the habit. I did not loosen anything, and I could say why the test was wrong.

What it costs

Running the old commit costs time. A cluster test is slow, and an A/B means building two binaries and running each many times.

It also needs a reproduction. Where a failure only appears under load, I load the machine until it fails often, as with the 16 busy processes above, then compare. A failure I can make happen 26 times in 30 is one I can fix.

What I learned

A timing flake and a real bug look the same from the outside: one red run, a green re-run. The difference shows up when you compare against an older commit. A real bug is usually a change in behaviour, and the old commit has the old behaviour.

A loosened bound doesn't just hide that. It teaches the test that the new behaviour is fine. Both the quorum refusal and the starved subscribers would be gone from the tests and still in the broker.

Trying it

docker compose up from the repo root starts a broker and a control plane. The cluster harness is in crates/testing/felix-cluster; task test runs the cluster tests. The delivery guarantees behind the subscriber queues are in the docs at docs.getfelix.dev.

docker compose up -d