Five weeks, 562 commits: Felix from 0.5.0 to 0.6.0
What we found, fixed and added between Felix 0.5.0 and 0.6.0: the bugs that shared one shape, the ways we found them, the features that apps asked for and why, and what building it taught us.
Notes from building Felix
How Felix works, why it is built the way it is, and what testing it against crashes and network splits has turned up.
Subscribe with RSSFast, without losing messages
How Felix answers a fsynced publish in a third of a millisecond, writes at the disk's full rate under load, and slows publishers down instead of dropping their messages.
Even overall, lopsided per stream
Felix placement kept every broker's leader count near the cluster-wide share, and a single stream could still land 1/1/10 with all its writes on one broker; this is how placement now spreads each stream's leaders too.
Before you loosen the timeout, run the old commit
Intermittent failures in Felix's cluster tests kept turning out to be real bugs in the broker, so before loosening a timeout I now run the test against a commit from before the suspect change; three cases show what that finds.
Acknowledged, then forgotten: a crash under periodic fsync
Under Felix's default periodic fsync, a crashed broker could come back missing records it had acknowledged while the cluster still counted it as caught up; this is how review and the TLA+ model found it on the follower and leader sides, and what the fix costs.
How we catch a 1% regression
Felix's rule is that a release must not be measurably slower than the one before it; this is how same-runner A/B runs, a Welch's t-test and throwaway bisect PRs in GitHub Actions find a 1-2% loss, and why a single benchmark run can't.