Five weeks, 562 commits: Felix from 0.5.0 to 0.6.0
What we found, fixed and added between Felix 0.5.0 and 0.6.0: the bugs that shared one shape, the ways we found them, the features that apps asked for and why, and what building it taught us.
Even overall, lopsided per stream
Felix placement kept every broker's leader count near the cluster-wide share, and a single stream could still land 1/1/10 with all its writes on one broker; this is how placement now spreads each stream's leaders too.
Before you loosen the timeout, run the old commit
Intermittent failures in Felix's cluster tests kept turning out to be real bugs in the broker, so before loosening a timeout I now run the test against a commit from before the suspect change; three cases show what that finds.
Acknowledged, then forgotten: a crash under periodic fsync
Under Felix's default periodic fsync, a crashed broker could come back missing records it had acknowledged while the cluster still counted it as caught up; this is how review and the TLA+ model found it on the follower and leader sides, and what the fix costs.
The follower that forgot
On an idle shard nothing ships, so a leader never learned that a follower had lost its disk; this is how a storage identity per data directory fixed that, and why it changes only when a directory may hold less than its peers were told.
Our model said everything passed. It was checking nothing.
Felix checks 107 TLA+ configurations in CI; this is about three times a passing check meant nothing, and the twin configurations, reachability checks and spec pairing we use to catch it.