Fast, without losing messages

A message broker has two jobs that pull against each other. It should answer quickly and move a lot of data, and it should never lose a message it was given. The easy way to look fast is to cut the second job short.

Felix does both with a handful of ideas, none of them new, each put where it does the most good. This post walks through them, with the numbers from one broker on Azure and where each number comes from.

Two kinds of latency

Ask how long a publish takes and the honest answer is: it depends on how many other publishes are waiting.

With nothing else going on, a Felix broker receives a 256-byte publish, writes it, fsyncs it to disk and acknowledges it in 0.31 ms at the median and 0.39 ms at p99. That's service time, the work itself. An in-memory stream skips the disk and answers in 0.16 to 0.18 ms.

Now load the same broker with a throughput test: 64 publishers, each keeping 64 batches in flight, each batch 64 records of 4 KiB, on a durable stream. The median ack takes 767 ms. The broker didn't get slower. The requests are standing in line.

Little's law says exactly how long the line takes. For anything that queues, L = λ × W: the number of requests in the system is the rate they complete times how long each one spends there. Turn it around and W = L ÷ λ.

Work it for that durable test. In flight: 64 publishers × 64 batches = 4,096 batches. The broker completes 1.40 GB/s, and a batch is 64 × 4 KiB = 256 KiB, so about 5,340 batches a second. 4,096 ÷ 5,340 is 0.77 seconds. The measured median is 767 ms. The same test on an in-memory stream completes 4.40 GB/s, so the law predicts 244 ms, and the median came in at 229 ms.

Nothing in that sum depends on how fast one request is served. Once the broker is busy, putting more in flight adds no throughput, only waiting. If you want lower latency at the same throughput, keep fewer requests in flight. And when a benchmark reports its saturated ack time as "request latency", it's really reporting how much its load generator chose to queue.

One fsync, everyone aboard

Durable writes are expensive because of fsync. It asks the drive to put the data on stable storage, and it's the slowest step on the write path. If every publish paid for its own, the broker would spend most of its time waiting on the drive.

But an fsync flushes the whole file, not one caller's bytes, so one flush can cover every write that's waiting. In Felix's storage layer every append that needs to be durable goes for the same flush lock. The one that gets it flushes. The others wake up, find their bytes already on disk, and return without flushing again. They're all acknowledged together.

PostgreSQL, MySQL and RocksDB do the same thing, under the same name. The more publishes are waiting, the more each fsync carries, so durable throughput climbs with load until the disk is the limit. On the test VM's NVMe volume, fio writes 1,414 MB/s with an fsync after every write. Felix wrote durable 4 KiB batches at 1.40 GB/s, fsynced before every ack, using less than half the broker's cores.

Fsyncing before the ack is FELIX_DURABLE_FSYNC_MODE=on_commit. The default is periodic, an fsync every 250 ms in the background, which survives a broker crash but can lose the last interval to a power cut. Pick on_commit when an ack has to mean "on disk".

Pacing instead of dropping

Sooner or later a broker gets more data than it can take. It then has two honest choices: slow the publisher down, or throw something away. Felix's ingress queue and byte budget are bounded, so one of the two has to happen.

With FELIX_PUB_INGRESS_WAIT=1, a fire-and-forget publish that finds the broker full waits for room instead of being dropped. It waits on its own, so the stream keeps reading the publishes behind it, up to 1,024 waiting per stream. Past that, the broker stops reading the stream, and QUIC flow control pushes back on the publisher. The publisher slows to the rate the broker can take, and nothing is lost. It's a metered on-ramp: cars wait at the light, and nobody gets pushed off the road.

That isn't the default for fire-and-forget. Out of the box, a fire-and-forget publish that arrives while the broker is full is dropped and counted in the broker's metrics, and the publisher isn't told. That's at-most-once delivery, which is all fire-and-forget promises. In the benchmark, the defaults accepted 4.2 GB/s and dropped 1.2 to 2.2% of what was sent.

Acked publishes never disappear quietly. On a durable stream the ack waits for the commit by default, and a publish that finds the broker full waits for room for up to 2 seconds before it's refused with an error. On an in-memory stream a full broker refuses it straight away. Either way the publisher knows, and decides whether to send again.

The fastest way to lose nothing turned out to be an acked publisher with a window. Each of the 64 publishers kept 64 requests in flight, and the broker took 4.4 GB/s of 4 KiB batches with nothing shed. The window keeps the broker busy, and the acks pace the publisher for free: when the broker slows down, so do the acks, and so does the publisher. Fire-and-forget with ingress wait was lossless too, but slower: 3.6 GB/s.

Straight to the owner

A Felix cluster splits each stream into shards, and each shard has one leader that writes it. A publish can go to any broker. If that broker doesn't lead the shard, it forwards the publish to the one that does. That always works, and it costs a hop.

ClusterClient skips the hop. The first time it publishes to a stream, it asks the cluster who leads each shard and keeps the answer. For every record it works out the shard from the key, with the same function the broker uses, and sends the record over its connection to that shard's leader. Refreshing the answer never holds up a publish. If leadership has moved and a cached leader turns a publish away, the client sends it once through the broker it first connected to, which forwards it.

On a three-broker cluster, forwarding added about 0.38 ms to an in-memory publish at the median: 0.83 ms through another broker, against 0.45 ms straight to the leader. Under load the hop costs much more. With two thirds of the publishes forwarded, an in-memory stream of 256 B batches ran at 864 MB/s; sent straight to the leaders, it ran at 2.0 GB/s. A replicated, durable stream fell further, from 475 MB/s to 107 MB/s, because each forwarded write holds its place on the link between brokers until a majority has it.

The numbers

One broker on an Azure Standard_L8as_v4 (8 vCPU, local NVMe in RAID0), over QUIC with TLS, with 64 publishers on four load generators. Throughput is the payload the broker accepted. These were measured on the final campaign build. The release build re-measured batched throughput within about 1% of them and latency within about 5%, except small fire-and-forget messages (#1469).

The NATS comparison ran on the same machines, interleaved, with NATS tuned first. With both systems fsyncing every write before acknowledging it, Felix was 1.8 to 3.5 times faster for single messages and for 4 KiB batches. For batches of small messages the gap was 14.8 times, but part of that is the benchmark tools: Felix kept 64 batches in flight where nats bench sends one at a time. Core NATS is faster at fire-and-forget, by 1.34 to 1.39 times. With no subscriber it checks for interest and drops the message, while Felix routes every record to its shard.

The full performance write-up has every condition, the trials behind each number, the cluster measurements and how to reproduce the run.

docker compose up -d