What an acknowledgement promises when a broker fails, and how those promises are tested. Each part links to the full write-up in the docs.
Clustering
Shards, replicas and failover
A stream is split into shards. The control plane gives each shard a leader broker and followers, and moves shards between brokers while they keep serving. What an acknowledgement promises is set per stream.
Control planeplaces shards on brokerspromotes shard 0 on broker 2broker 1 rejoinsmoves shard 0 to broker 1
One stream, three shards, three brokers. Every shard has a copy on each broker, and each broker leads one of them.
What an acknowledgement promises
Leader
The shard's leader answers alone.
Acknowledged
On a durable stream, when the leader has written the record to its log. On an in-memory stream, when it has queued it.
Leader lost
Records it had not yet shipped are lost. With FELIX_ACK_ON_COMMIT=0, so is a record still queued when it crashes or its lease runs out. Replication lag bounds that window and is exported as a metric.
Failover
Roughly 13 to 26 s after a crash with the defaults. A broker stopped gracefully hands its shards to followers first.
Closing the loss window would take a majority round on every write, which is what Quorum does.
Quorum
A majority of the replicas answers.
Acknowledged
When a majority of the replica set holds it, the leader counted: 2 of 3, 3 of 5.
Leader lost
Nothing acknowledged is lost. Only a replica holding every acknowledged record can be promoted.
Failover
About 5 s on a durable stream, on its followers' word, once the generation_start, majority_ack and lease_free_reads fleet features are finalized. A new cluster finalizes them when its first broker registers; an upgraded one waits for the operator. Otherwise on the lease.
With no majority reachable, a publish fails after 5 s with quorum_timeout. The record may still survive, so the outcome is unknown, not refused.
Durability, per stream
In memory, the default: never on disk and not replicated. Delivery is at most once, and records go with their leader.
durable: true: a checksummed segment log, replicated to the shard's followers and replayable from any retained offset. Delivery is at least once.
The broker's fsync mode says when a write is on disk: on_commit before the ack, periodic (the default) within 250 ms, none when the OS gets to it. By default an acknowledged record is already in the log file, so it survives a broker crash; under periodic a machine crash can lose the last interval.
With replication_factor: 1, the default, a durable shard has no follower and waits for its broker to return.
Ordering
Per shard, so per routing key. A record with no key goes to shard 0.
A durable stream with one shard is totally ordered.
Nothing is ordered across streams, and on an in-memory stream two concurrent publishers can be seen in either order.
Delivery
Each event on a durable stream carries its offset. Subscribe from earliest, latest or an offset.
Under the default overflow policy, a slow subscriber is dropped from, never waited on. On a durable stream its subscription ends at the first drop and names the offset to resume from.
Under overload, with the defaults, a broker sheds fire-and-forget publishes. An acknowledged publish it has no room for is refused with a retryable overloaded error: at once on an in-memory stream, or on a durable stream (acked on commit, the default) after waiting up to 2 seconds for room (FELIX_PUBLISH_QUEUE_WAIT_MS). With pub_ingress_wait, fire-and-forget publishers wait instead.
Idempotent producers make a re-sent publish land once. Exactly-once delivery is not offered.
Partial The control plane decides every failover, so while it is down a dead leader is not replaced. Read the full contract
How Felix is tested
Tested against crashes, partitions and power loss
The replication claims above rest on evidence anyone can rerun: models that search every interleaving within their bounds, a checker that drives a real cluster through faults, and storage tests that rebuild the disk a power cut could leave after each flush.
campaign nightlyfaults 20 minstreams Quorumseed new each night
A timeline of one illustrative night. Faults strike a three-broker cluster and its control plane one after another: broker 2 is killed, broker 3 is frozen, broker 1 is partitioned from its peers, broker 3's outgoing messages are dropped, broker 2's clock runs twenty times fast, every broker loses power at once, and the control plane crashes in the middle of a shard move. Each fault heals before the next. Below, acknowledged records keep arriving except while the partitioned leader and the powered-off cluster cannot reach a majority, where two writes end with an unknown outcome. After the last fault heals, every stream is read back and each acknowledged record is found in place: none lost or reordered.control plane
7crash mid-move
broker 1
3partition
broker 2
1kill
5clock 20×
broker 3
2freeze
4one-way link
6power lossevery broker at once
acknowledgedacked
acknowledgedoutcome unknown: may land or not, and either is allowed
1kill broker 2
2freeze broker 3
3partition broker 1
4one-way link broker 3
5clock 20× broker 2
6power loss every broker
7crash mid-move control plane
Faults in progress. Nothing is judged until every fault has healed.
Final read. Every stream is read back from its leader and compared with each acknowledgement.
0
acknowledged records lost or reorderedin the final read, taken once the last fault healed
No acknowledged write is missing from the final read.
A write acknowledged before another was sent sits earlier.
No value sits at two offsets.
Every read agrees with the final log where they overlap.
A write refused outright never appears.
Subscribers get offsets in order, with every record or a visible gap.
An illustration of the nightly history campaign, not a recorded run. Each night runs 20 minutes of single faults on three brokers and 20 minutes of compound ones, such as the power loss and the control-plane crash, on four. Each uses a new seed, which a failing night prints so it can be replayed.
TLA+ models
Replication, promotion and shard moves are specified in TLA+. TLC checks every interleaving within small bounds, and random walks sample behaviours hundreds of steps long. The model found a way to lose an acknowledged Quorum record that the design's prose had missed, and the broker was fixed.
every code changewalks nightly
History checker
Clients write to Quorum streams and a Quorum cache on a real cluster while faults strike: kill, freeze, partition, one-way links, clock faults, failed fsyncs, shard moves, power loss of every broker and a control-plane crash mid-move. Every operation is recorded and checked against eleven rules.
every pull request20 min nightly
Power-loss sweeps
The storage engine runs a workload while a test layer records what each flush made durable, then rebuilds the disk a reboot could find, with unsynced writes dropped, torn or zeroed. Every record acknowledged as durable must come back, with no gap before it.
every pull request110 seeds nightly
Recovery rules
A torn tail is repaired: a record a crash left unfinished is cut away.
Corruption anywhere else refuses to start, naming the segment and byte position.
Indexes are rebuilt from their segments, never trusted.
Client conformance
A catalogue of scenarios every client must pass, with the Rust client as the reference. CI checks the Python and TypeScript clients against it and drives the C interface against the same fixture. A cluster suite checks that a client can't tell one broker from three.
every code change
Fuzzed decoders
The decoders for wire frames, Kafka requests and record batches, and the on-disk records, indexes and recovery state are fuzzed. A short run gates each pull request, and a nightly campaign keeps building its corpus.
