Throughput and latency
One Felix broker accepts 4.4 GB/s with every message acknowledged, writes 1.4 GB/s fsynced to disk, and acknowledges a single fsynced publish in 0.31 ms — about a third of a millisecond. Three 4-core brokers keep three fsynced copies of every durable record at 656 MB/s. Every number here comes from the v0.6.0 performance write-up, which has the exact conditions and how to reproduce them.
Measured on one broker
Felix 0.6.0 on an Azure Standard_L8as_v4 broker (8 vCPU, local NVMe), driven by 4 load generators in the same region over QUIC with TLS. "Durable" means the broker fsyncs before it acks. These numbers come from the final campaign build; the release build re-measured batched throughput within about 1% of them, and latency within about 5%, except small fire-and-forget messages (#1469). Each card links to the table it comes from.
4.4GB/s
Acked, nothing lost
4 KiB messages in batches of 64, in memory. Every publish acknowledged and nothing shed: 1.07 million messages a second.
Lossless throughput1.4GB/s
- Felix, fsynced
- 1,399MB/s
- Raw disk,
fio - 1,414MB/s
99% of what the disk can do with an fsync per write
Durable, fsync before every ack
Each 4 KiB batch of 64 is fsynced to NVMe before the ack. The disk itself manages 1,414 MB/s with an fsync per write (fio), so Felix is limited by the disk, not the broker.
- p50
- 0.31ms
- p99
- 0.39ms
- p999
- 0.44ms
One fsynced publish, acknowledged
A 256 B durable publish, fsynced before the ack, with one in flight. Even the slowest in a thousand stays under half a millisecond. In memory, p50 is 0.16 ms.
Unloaded latency8.78M messages/s
Small messages, acked
256 B messages in batches of 64, in memory, every batch acknowledged: 2.2 GB/s of payload. Batching lifts the small-message rate 22 times over one message per request.
Throughput by shape1.8–3.5×vs JetStream
Faster on durable writes
Against NATS JetStream on the same VMs, both fsyncing every write before the ack, for single messages and for 4 KiB batches. Core NATS is faster at fire-and-forget.
Against NATS JetStreamLatency, one publish in flight
256 B messages, median of three runs. The solid bar is p50 and the tint runs to p99. Delivery is when a subscriber on the stream receives the message; a durable message is delivered only once it is durable.
The fsync costs about 0.13 ms at 256 B. The tail stays tight: p999 is within 50% of p50 in every row. Unloaded latency has 4 KiB too.
Acked throughput by shape
Batches of 64 lift 256 B messages from 102 MB/s to 2.2 GB/s in memory, and durable 4 KiB and 64 KiB batches run at the disk's fsync rate. Throughput by shape has every cell.
Fast when idle, predictable when saturated
With few publishes in flight, Felix acks in well under a millisecond. Push it to full throughput with thousands outstanding and acks wait in line for their turn: the wait is the work outstanding divided by the rate it completes, as Little's law predicts, and the measured p50s land on that prediction. Keep few requests in flight for sub-millisecond acks, or many for throughput. Two kinds of latency works through the numbers.
Ack latency against publishes outstanding
With one publish outstanding, latency is service time: 0.16 to 0.38 ms. Every point to the right is queueing, and sits close to the cross that Little's law predicts from throughput alone.
Against NATS JetStream
NATS ran tuned, on the same VMs, with the same 64 publishers and payloads, interleaved with Felix. With both fsyncing every write before the ack, Felix is 1.8 to 3.5 times faster for single messages and for 4 KiB batches, and 14.8 times for batches of 256 B messages. With periodic fsync on both sides it is 1.25 times faster at 4 KiB and 9.5 times at 256 B, and 1.2 times in memory. Core NATS is 1.34 to 1.39 times faster at fire-and-forget, where it discards messages nobody is subscribed to.
The two systems are built differently, so no pair is an exact match. Read each ratio with where it still differs. The 14.8× durable batch pair is partly the benchmark tool: nats bench sends one atomic batch at a time, while Felix keeps 64 in flight. Full results are in Felix against NATS JetStream.
Felix rate divided by NATS rate
Ten pairs, payload at the server, log scale centred on 1×. Each bar names the pair's main caveat.
Three brokers, three copies
On a durable Quorum stream with a replication factor of 3, every record is written to all three brokers and each broker fsyncs its copy. The publisher gets its ack once the shard's leader and at least one follower have fsynced, so an acknowledged record is already on two disks, and the third copy follows. Each stream's shard leaders are spread evenly over the brokers, so every broker carries its share of the writes. Each card links to the table it comes from.
Felix 0.6.0 release build on three Azure Standard_L4as_v4 brokers (4 vCPU, local NVMe), one in each availability zone, driven by 4 load generators over QUIC with TLS.
656MB/s
- Each broker appends
- 661MB/s
- Its raw disk,
fio - 731MB/s
90% of each broker's disk
Durable, three fsynced copies
4 KiB batches of 64. Every broker stores every record, so each disk writes the whole stream; a write is acknowledged once two of the three have fsynced it. 256 B batches of 64 run at 475 MB/s, 1.86 million messages a second.
Cluster throughput4.6GB/s
In memory, across the cluster
4 KiB batches of 64, acked. An in-memory stream keeps one copy of each record, on its shard's leader, and the shards are spread over the three brokers. Three 4-core brokers move more than the single 8-core broker's 4.4 GB/s.
Cluster throughput4.4×with ClusterClient
Straight to the right broker
ClusterClient sends each publish to its shard's owner. Through any broker, two in three publishes take an extra hop: durable 256 B batches run 4.4 times slower and in-memory ones 2.4 times, and one in-memory publish in flight acks in 0.83 ms instead of 0.45 ms.
Straight to the owner against through any broker
Payload with batches of 64, 64 in flight per publisher. A forwarded publish is decrypted and re-encrypted at every hop, and a durable one holds its shard until the quorum commits. ClusterClient avoids both. Forwarding has the numbers.
