Five weeks, 562 commits: Felix from 0.5.0 to 0.6.0
What we found, fixed and added between Felix 0.5.0 and 0.6.0: the bugs that shared one shape, the ways we found them, the features that apps asked for and why, and what building it taught us.
Before you loosen the timeout, run the old commit
Intermittent failures in Felix's cluster tests kept turning out to be real bugs in the broker, so before loosening a timeout I now run the test against a commit from before the suspect change; three cases show what that finds.
Acknowledged, then forgotten: a crash under periodic fsync
Under Felix's default periodic fsync, a crashed broker could come back missing records it had acknowledged while the cluster still counted it as caught up; this is how review and the TLA+ model found it on the follower and leader sides, and what the fix costs.
How we catch a 1% regression
Felix's rule is that a release must not be measurably slower than the one before it; this is how same-runner A/B runs, a Welch's t-test and throwaway bisect PRs in GitHub Actions find a 1-2% loss, and why a single benchmark run can't.
Our model said everything passed. It was checking nothing.
Felix checks 107 TLA+ configurations in CI; this is about three times a passing check meant nothing, and the twin configurations, reachability checks and spec pairing we use to catch it.