Five weeks, 562 commits: Felix from 0.5.0 to 0.6.0

Felix 0.5.0 was tagged on 19 September 2026. 0.6.0 was tagged on 23 October, after nine previews. Between the two there are 562 commits and 192 closed tickets, and a CHANGELOG section that runs to about four thousand lines. Nobody should have to read that to learn what happened.

This post is the map. It groups the work into six threads, says how each kind of bug was found, what the bugs had in common, and what we added on purpose and why. Several of the stories already have a post of their own. Where one does, we say what it found in a sentence or two and link to it, rather than telling it again.

How we found them

The tickets did not come from one place. We counted where each one said it came from, and Figure 2 has six sources.

Models. The docs/formal directory held one TLA+ module and five configurations at 0.5.0. At 0.6.0 it holds seven modules and 140 configurations. Our model said everything passed. It was checking nothing is about the discipline that growth needed. A model finds bugs the tests never reach, and it can also drift from the code. We come back to that below.

History campaigns. A harness that starts a real cluster, injects faults at random, records every acknowledgement and every delivery, and then checks the whole record. When it fails it prints a seed, and the seed reruns the same faults.

Power-loss sweeps. A nightly run that simulates a machine losing power at many points in a write, over 110 seeds per scenario in the run that found one of the bugs below (#1218).

Reviews. Repeated architecture reviews, and an external one, each of which filed its findings as tickets. Twenty-eight of the 192 cite one.

Performance runs. Load on real cloud VMs. They were meant to measure speed, and they kept finding correctness bugs and honesty bugs along the way.

The apps. Felix has companion apps: a drawing canvas, an arena game, a city simulation, a webhook relay, a gateway, a block builder and an agent swarm. Thirty-eight of the 192 tickets name one, and the canvas alone is named in 27. These are tickets that begin "while building X, we hit...".

The sixth source is the one nobody builds on purpose: a test that failed once in CI. Eleven tickets mention a failed or flaky CI run, and several turned out to be real. Before you loosen the timeout, run the old commit is the rule we settled on.

Thread one: a name that outlived what it vouched for

The largest named area in Figure 1 is replication and placement, and most of what is in it has one shape.

Felix decides failover, quorum and promotion by consulting a claim. A follower is reported caught up. A fence collected a majority. A Raft member holds a vote. A name refers to a stream. Each claim was true when it was made. The bug each time was that the world changed and the claim stayed.

Some of these have a post:

• A leader on an idle shard kept reporting a follower as caught up after the follower lost its disk, because nothing ships on an idle shard and nothing looked again. The follower that forgot is how a storage identity per data directory fixed it (#1266).
• A broker that crashed under periodic fsync came back with a shorter log while the cluster still counted it as caught up, on both the follower and the leader side. Acknowledged, then forgotten is how review and the model found both sides (#1284, #1295).
• A deleted stream and a new stream with the same name could be the same stream to a broker's disk, so the old records came back. The data came back is the incarnation that tells them apart, and the rollout that kept a mixed-version control plane from disagreeing (#1208).
• An idempotent producer's sequence did not survive a leader change, which Point your Kafka client at Felix explains from the Kafka side (#608).

The ones that have no post of their own are as instructive.

A wiped Raft member voted. The control plane can keep its metadata in an embedded Raft group. A chaos test failed in CI with "instance 1 lost acknowledged write". The test had failed 0 of 15 runs under light load, so it was easy to blame, but the ticket worked out why it was intermittent and then wrote five steps that lose a write every time: kill A, write a record that L and B acknowledge, pause B, wipe L and restart it, restart A. A wins an election with the wiped L's vote, and neither has the record. A member that has lost its state had restarted as a voter, and Raft's safety rests on a voter never forgetting its vote or its log (#667). The fix: a member that starts with no Raft state refuses vote requests and does not stand for election. It asks its peers whether the group exists and follows the leader without a vote until it has applied the leader’s last log index (#670).

A fence that counted the wrong set. When a follower is promoted it fences a majority of the replicas before it serves. We counted the majority of the new replica set, and a failover rebuilds that set from live brokers. With three copies and two lost, the new set could be the promoted broker alone, which is a majority of itself, and it opened without a write that the old leader and another follower had acknowledged. An external review found it and our own design pass confirmed it. The ticket carried a standing instruction: do not finalize majority_ack until this is fixed (#840). A durable Quorum shard now keeps its previous replica set, the dead leader in it, so the fence needs a majority of the set that acknowledged.

A deposed leader that came back. A history campaign found one value at two adjacent offsets in the final log, and a live subscriber told that the first offset held no event. A broker that had drained a shard into a move kept its draining flag after the move finished. When a later failover gave it the shard back, it took that for a cancelled move’s hand-back and opened without fencing. The old leader, cut off rather than gone, could still commit with a follower that had not heard of the promotion (#971). The shape is the same. The flag outlived the move. Now only the generation right after the draining one counts as a hand-back (#987).

We take one lesson from the list, and it is a cheap one to apply. When a decision rests on a claim, write down what evidence the claim came from and what can invalidate it, and make the invalidating event clear the claim. The storage identity, the incarnation, the previous replica set and the vote-withholding rule are all that, in different clothes.

Thread two: what survives the power going out

Storage is a log-structured segment store. Records are never rewritten, a torn tail is repaired and interior corruption is fatal, so that recovery can refuse to start rather than lose an acknowledged record quietly. Those are strong rules, and most of this thread was finding places where the code did not keep them.

The nightly power-loss sweep found the sharpest one. A background roll seals a full segment while flushes continue. If the seal finished after a flush had already covered the next segment, it set the durable mark back to that segment's header, and on an idle shard nothing moved it forward again. After a power loss, rot in acknowledged records of that segment was then repaired as a torn tail instead of being refused (#1220). A related race let a seal put back the length a truncation had cut (#1222). Both needed a crash at one moment in a race between two threads, which is what a sweep over many seeds is for.

Others in the same family:

• A power loss on a fresh node could drop a store root and the acknowledged records in it, because the directory holding the root was never synced into its parent (#888).
• With both followers of a shard down, a tight retention bound deleted records only the leader had, including ones that publishes were still waiting on, and the followers' answers later moved the quorum mark over the deleted records. Retention now stops at the commit offset on every replica of a Quorum shard, and a stream whose followers are down grows past its bound until a majority is back (#1094).
• A full disk while creating a log left it reporting corruption on every open (#983).

And one that points the other way. The sweep also failed because its own model was wrong. When two flushes of one file overlapped, the one that read first could record its read last, so the test's picture of the device held less than the real device would, and the sweep built a crash that no real disk could produce (#1218). A harness is code, and it is as likely to lie as the thing it tests. We come back to that in the lessons.

Thread three: a slow consumer, and the memory it can pin

Felix gives every subscriber its own bounded queue and, by default, drops a batch for a subscriber that falls behind rather than slow a publisher. One slow subscriber shouldn't slow anyone else is the design and the 2% regression we traced to where the feeder gave a byte budget back.

The bound was the first problem. The queue was bounded by entry count, and an entry is a whole publish batch, which can be as large as the 16 MiB frame limit. The ticket did the arithmetic: 4,096 queued batches of 16 MiB is 64 GiB held by one stalled consumer, before the overflow policy ever fires. Small-payload workloads never notice. Large-payload ones can run a broker out of memory through one reader (#1157). Each subscriber’s queue is now bounded by bytes as well, 128 MiB by default (FELIX_SUB_QUEUE_MAX_BYTES), and the frames waiting in the writer lanes of a connection have their own per-connection bound of 256 MiB (FELIX_SUB_EGRESS_MAX_BYTES) (#1189, #1195). The same gap existed in cache watches and the replay ring (#1201), and under the block policy one full subscriber held up a batch's fanout to everyone else (#1179).

The publisher side had the mirror-image problem. A client hashed every shard of a stream onto one pooled QUIC stream, which answers in request order, so one stuck shard held back answers other shards had already committed (#843). Later, in the performance runs, a full ingress queue stalled the whole connection instead of the one stream that was waiting (#1416). A publish that cannot be queued now waits off the read loop, and the connection keeps serving its other streams.

The two sides have the same cure. A bound should count what costs the resource, which is bytes here and not entries. And a wait should belong to the thing that is waiting: one subscriber, one shard, one stream.

Thread four: speed, and being honest about it

The performance push started from measurements that were not flattering. These come from the tickets that opened the work, taken in the campaign’s own runs on cloud VMs, and are what each investigation measured at that point, not release results. Durable ingest with an fsync per commit reached about 82% of what the NVMe device could do (#547). A routing key cost about 30% throughput because it forced JSON framing (#549), which led to retiring JSON from the data path (#550). Twelve shards on one broker measured the same as one (#552), because the ceiling was a single endpoint task on one core (#557), so a broker can now bind several listeners and binds up to four by default (#720). Ingest copied each payload byte twice (#902). Durable appends called write on async worker threads, where a slow disk could stall the network endpoint (#909).

Our favourite is a latency bug. In one of those runs, a subscriber on an otherwise idle stream saw events about 5.8 ms after a publish was acknowledged in 175 µs. The batching delay in the feeder restarted on every message, so a steady trickle never let it expire, and a subscriber waited for 64 events (#719). The delay now bounds the whole batch.

The published 0.6.0 numbers, from the performance page, are the release results. They are not a sum of the tickets above. One broker accepts 4.4 GB/s with every message acknowledged, and writes 1.4 GB/s fsynced to disk against 1.414 GB/s that the raw disk manages with an fsync per write. A single fsynced publish is acknowledged in 0.31 ms. Three 4-core brokers keep three fsynced copies of every durable record at 656 MB/s. Fast, without losing messages tells the group-commit and pacing story, How we catch a 1% regression is how we guard those numbers, and Ten parcels, one envelope is the UDP offload bug a bigger MTU exposed.

Two of the late findings are worth telling.

A thundering herd. With thousands of publishes in flight against a full ingress queue, each freed slot woke every blocked publisher to contend for it. A freed slot now admits one waiting publisher, in arrival order. A deeper default queue, 512 instead of 64, took 256-byte unbatched acked publishing from 95 to 163 MB/s with 64 publisher connections and spread keys, in the run the CHANGELOG records (#1406); that is a measurement of that change, not a release figure.

A deadlock between brokers. In an RF=3 run, no publish succeeded for 30 seconds. Brokers forward publishes to each shard's owner. The owner held a peer stream for the whole quorum wait, and replication shared that stream's lanes, so two brokers forwarding to each other each queued their replication behind the other's forwards, and neither quorum mark could move until a timeout fired. The cluster recovered five minutes after the load stopped (#1458). Clients that publish owner-direct never hit it. It showed up in a 90-second load cell.

The honesty bugs came from the harness. The harness read broker CPU from the first iteration of top, which has nothing to difference against, and printed an idle broker that was appending 58.9 GB (#537). Session results were gitignored, so a published number had no evidence anyone could audit (#539). Session evidence is committed now.

Thread five: the catalog, and a cursor that claimed too much

Brokers learn about tenants, streams, caches and shard assignments from the control plane, by taking a snapshot and then following a feed of changes. This looked boring and was not.

Reading another change, we found three ways the feed could silently skip a change (#1224). A snapshot read its rows and then, in a separate statement, the sequence number to follow from, so a delete landing between the two was counted but never seen. Sequence numbers came from a BIGSERIAL, so they were handed out in allocation order and not commit order: a reader could pass a change that was still committing. And a page of changes cut by the page limit returned the sequence number after the tail of the whole feed, not after its last row, so a broker more than 1,000 changes behind skipped everything past the first page.

None of these came from a test. They are what you find by reading a cursor and asking whether the number it hands back is the number it earned. They also made the incarnation work riskier, because a broker that misses a delete and a create serves the deleted stream and refuses the new one's replication. The fix reads rows and sequence at one point, allocates in commit order, and answers a full page with the sequence after its last change. Brokers now follow a feed page by page within a sync pass.

The same family: a control-plane instance reported ready before its replicated state was current (#601), a metadata version gate covered new commands but not new fields on old ones (#842), and a Raft leader retried a failed member in a tight loop (#945). The gate is what lets a mixed-version control plane roll forward without storing different answers, and The data came back shows it at work.

Thread six: authorization that only ran once

The auth work was mostly hardening, and the CHANGELOG ranks it under Security rather than Fixed. A review of the token exchange and refresh path, the surface an earlier review had not covered, opened the thread (#524). It continued with tenant administrators who could remap claims on a shared identity provider (#845), and a broker's rotated refresh token written world-readable, mode 0644 under the usual umask (#913).

Two findings made the same point. A revoked user kept every stream it already had open until its token expired, because authorization was checked when a stream opened and never again (#1132). Revoking a principal, or removing a policy or a role assignment, now ends what its tokens opened, on every broker within about one control-plane sync interval, with a revoked error. And a revoked gateway could keep delegating with a control-plane token it had received before the revocation, because delegation checked the subject's issue time against the revocation cutoff and trusted the gateway's own claims (#1205). An external review found that one.

A valid token is a claim about the past. What the broker needs to know is whether the principal is still allowed. That is the same sentence as the first thread.

What the bugs had in common

Three things came up often enough to name.

A bound in the wrong unit. Entries instead of bytes, a connection instead of a stream, a cluster total instead of a per-stream spread. Even overall, lopsided per stream is the placement version.

The checker was also wrong. The model said Felix lost acknowledged records, for a design the code had stopped having, because the code gained a report-before-mark ordering and the model did not. CI pinned that as the expected result, and a ticket was filed for a large change the code did not need (#598). The power-loss model mixed up a reused inode with the file it replaced (#1221). A history workload kept a dead connection and reported only nine cache puts acknowledged (#1223). One history failure was real, and it was in the client: a dropped idempotent producer kept sending what it had been handed, so an append recorded as unknown could land after the final read (#1075). We added a check that pairs spec changes with code changes, and extended it when the first version missed the storage-identity code (#1337). Every harness needs a way to be caught lying, and the performance harness that reported an idle broker under load is the same lesson.

A failed run that went away was worth chasing. The wiped Raft voter and the double offset each first showed up as one failed CI run, and each was a real bug, in the control plane and the broker. A third, a record delivered past the log end, looked the same and turned out to be client behaviour, which is still a defect worth fixing.

What we added, and why

Features in this release mostly did not start as ideas. They started as a workaround in someone's code.

Acknowledged now means committed

The performance pages described durable publishing as "the broker fsyncs before it acks". That was true only with FELIX_ACK_ON_COMMIT=1. By default the broker acknowledged a publish when it queued it, so a broker crash or a lapsed lease could lose a record the client had been told was stored. Someone who created a durable stream and kept the defaults had a weaker guarantee than the documentation said (#1421).

We closed the gap in the safe direction. A durable stream now acknowledges after commit by default, and acks carry the offset. What "commit" buys depends on the fsync mode. With the default periodic mode the record is in the log file and survives a broker crash, and a machine crash can lose the last 250 ms. With on_commit the device has flushed it. FELIX_ACK_ON_COMMIT=0 restores the old behaviour, and in-memory streams still answer when the publish is queued. Acked publishes now cost a write's latency, and the release notes say so.

A new cluster starts with its safety features on

Cross-broker behaviour is gated behind fleet features so a cluster of mixed versions does not disagree about what a record means. An operator finalizes each feature once every broker supports it. That is the right default for an upgrade and the wrong one for a brand-new cluster, which ran for a while on the weaker mode: a fresh RF=3 cluster acknowledged Quorum writes on the control plane's replica report until someone finalized majority_ack.

A new cluster now finalizes every fleet feature its brokers report at the first broker registration, through the same checks as an operator's finalize. A cluster that is upgraded still waits for the operator. This is why the thread-one fixes matter in practice: the safe behaviour is what a fresh cluster runs.

Clients that go straight to the owner

A plain publish to the wrong broker was forwarded, and the client never learned who owned the shard, so it was forwarded for as long as it ran (#536). ClusterClient now looks up shard leaders from the cluster on a stream's first publish, every few seconds and after a publish finds one wrong. It also publishes batches owner-direct, splitting records by shard, sending each shard's ordered batch to its leader and reporting failures per shard. The deadlock above is why this matters beyond speed: an owner-direct publish takes no forwarding hop.

Asking a stream about time

Every record was stored with an append time that the API never exposed. The webhook relay wanted to replay "from time T", so it binary-searched the stream by subscribing at a midpoint offset and reading one record, about 24 subscribes for 10 million records (#975). Events and group records can now carry their append time, behind a negotiated flag so older peers exchange the same bytes, and ClusterClient::offset_for_time finds the offset for a time on one shard, asking whichever broker leads it and following redirects (#1453). Replicas now keep the time of the broker that led when the record was written, so a record's time no longer shifts after a failover (#1045).

What the other apps asked for

A short list, each with the ticket that holds the reason: a conditional cache put, because the relay needed compare-and-set (#976); a conditional append that writes only at the offset its writer expects, because the arena needed writer fencing (#1017); a bounded range read, because replay ran on into live delivery (#1018); delivered events that say who published them (#1010); a subscriber that chooses its own queue capacity and overflow policy (#1019); consumer groups with claim control, reset and a reported cursor (#973, #974); and one client connection that acts for many users, because a gateway otherwise needs a client per user (#969, #1129).

Moving, placing and trying

Shards move online while their leaders are alive (#130), placement spreads a shard's copies across zones, and a stream created with a region stays in it. A broker hands its shards off before it stops. Kafka producers and consumers can use durable streams, which the Kafka post explains. felixctl is a command-line tool for all of it (#872). And docker compose up brings up a control plane, a broker and development credentials, so that felixctl pub and felixctl sub work from a fresh clone (#1345). We wanted a person to reach a first message in one command, and we run that command in CI so it keeps working.

Right after 0.6.0

Four fixes that came out of using 0.6.0 followed it in a follow-up release. A write can name its shard instead of hashing a key, because the city simulation had been salting keys until each district landed on its own shard (#1465). A subscribe now joins at the latest record before trailing generation-start markers, after the canvas hung on a log that ends in one (#1476). Unacked publishes parked per stream are limited, which fixes the 256 B fire-and-forget slowdown in the known issue (#1477). And a busy follower's late answer now counts toward the quorum mark, which removes the stalls of about 1.8 seconds that appeared under load once a new cluster finalized its fleet features (#1481).

What the release taught us

Write the invalidation, not just the claim. Every decision here rested on a claim: a storage id, a name, a vote, a fence’s majority, a cursor, a token. Each was true when made, and the bug was that it stayed after the world changed. A good test is to ask of any cached or reported fact: what event makes this false, and who is told?

Models earn their place twice. Once by finding what tests cannot, such as the leader-side crash loss, and once by being compared with the code. A model that has drifted is worse than none, because CI will defend it. We pair spec changes with code changes now, and when a counterexample appears we fix the code.

Rerun it against the old commit. Before we change a timeout we run the test against a commit from before the suspect change, as the post on it describes.

Defaults are promises. The fsync-before-ack documentation and the default disagreed, and no test could notice. When we write "durable", we now check which setting makes it true.

Build things on it. Thirty-eight of the 192 tickets name an app we built on Felix. An application finds the gap between "the API exists" and "the API is usable" faster than any review.

Say what you measured. The 0.6.0 notes list a known issue: fire-and-forget batches of 256 B messages ran about 13% slower than the build the performance runs measured (#1469). We found it by comparing a release build against the measured one, and we wrote it down. The fix followed in the release after 0.6.0 (#1477).

Where Felix is heading

We want an application not to need a broker, a cache and a job queue beside each other, with three clients, three sets of credentials and three failure modes, plus glue to keep them agreeing. Felix is a replicated log that is read forward as a stream, as its latest value per key as a cache, and as a durable cursor as a queue. The aim is for events, state and work to be one system, so that the thing which tells you a record was acknowledged is also the thing that tells you the state it changed and the job it queued.

The companion apps show the kind of application we have in mind: a shared drawing canvas with live cursors and history you can scrub, an arena game whose every match is a log you can replay, a city simulation you can scrub back through, a swarm of workers that come and go, and a webhook relay that takes in events durably and replays them after an outage. What they share is state that changes fast, many participants who must agree on it, and a past that people want to revisit. We want building and operating those to take much less code and fewer moving parts than it does today.

That points at the harder half of Felix's stated purpose: coordination-store semantics, where you read the current value and then receive every change with no gap, at the fanout of a messaging system. Log-backed caches do this already, and the status tables on the docs site say plainly where the claim stops. We would like fewer rows there to say "partial".

We will not trade three things for any of that. Tail latency should be predictable, which is why the same performance page that gives a throughput figure gives a p999. An acknowledged record should not be lost, which is the whole of threads one and two. And a slow consumer should hurt only itself. When a feature would make one of those conditional, the feature waits.

For someone betting on Felix, we mean 1.0 to be a promise about stability that is backed by evidence. Felix is pre-1.0 and its APIs can still change between releases. What we want to be able to say is that an upgrade rolls through a cluster without losing anything, that the wire format and client APIs are stable, and that a long-running real workload has been running on it. The fleet-feature gates and the fresh-cluster default in this release are a step toward the first of those. We have said what we want and have not said when.

The direction is visible in the open tickets, and also in the apps' gaps. There are open investigations into position tokens for read-your-writes (#1384), a version precondition on a commit (#1051), exactly-once processing on a single shard (#1380), failover that no longer waits on the control plane's lease (#1009) and a cache key index that does not live wholly in memory (#1073). They are open questions and not commitments.

Trying it

The quickest way to see any of this is the one-command stack from the README: docker compose up and then felixctl pub orders 'hello'. The status table says what each capability does today. The CHANGELOG has the whole 0.6.0 section, with the upgrade notes first: the commit-ack default, the new queue depth, and the metadata levels a Raft control plane needs before the recreated-stream fix applies. The tickets linked above carry the reproductions.

docker compose up -d