One slow subscriber shouldn't slow anyone else

A stream in Felix can have one subscriber or a thousand. Sooner or later one of them stops reading. Its laptop goes to sleep, its render loop hangs, its network link degrades.

That reader is its own problem. It shouldn't slow the publisher, and it shouldn't slow the other 999 readers. By default, in Felix, it can't.

Tightening that guarantee once cost 2% of throughput. Finding it took a day of benchmarks in CI, and the fix was moving one semaphore release a few lines later in the same loop.

What isolation means here

Each subscriber gets its own bounded queue on the broker. When a publish lands, the broker tries to put it in every subscriber's queue. If a queue is full, the overflow policy decides what happens.

The default is drop_new. The arriving batch is dropped for that one subscriber, the drop is counted, and the broker moves on to the next queue. The publisher never waits for a reader.

The other option is block. Under block, the broker waits for room instead of dropping. Nothing is lost, but a subscriber that stops reading holds every publisher on the shard, with no timeout, until it reads again or disconnects. That's a fine trade for some workloads. It's a bad default, because one stalled reader turns into back-pressure on every producer of the stream.

Why it matters

A dropped record is only acceptable if you can see it. On durable streams, every delivered event carries its log offset. Offsets are dense, so a jump from 1041 to 1047 means five records were dropped for you. A gap in offsets is exactly a drop.

A client that negotiates it gets more than the gap. The broker ends the subscription at its first drop with subscription_lagged, naming the offset to resume from. The client reads the missing range from the log and carries on. So drop_new doesn't mean “lose data quietly”. It means the slow reader catches up from disk instead of making everyone else wait for it.

How it works

There's more between a publish and a subscriber's socket than one queue, and each stage needed its own bound.

Encode once

When a publish is durable, the broker wraps the batch in one DeliveryEnvelope and puts a pointer to it in every subscriber's queue. The envelope caches its encoded frame. The first subscriber to need the frame encodes it, and everyone after that reuses the bytes. A published batch is encoded once whether ten subscribers read it or ten thousand.

There's one wrinkle. Subscribers don't all ask for the same frame. Some negotiated event offsets and some didn't, so the envelope keeps one cached frame per shape. That's still a handful of encodes per publish, not one per reader.

Feeders and writer lanes

Each subscription has a feeder task. It takes batches off the subscriber's queue, coalesces small ones into one frame, and hands the frame to a writer lane. The lanes are shared by the subscriptions on a connection, four by default, and they do the actual QUIC writes.

The lane hand-offs are bounded by count and wait when full, but that wait backs up into the subscriber's own queue. That's where drop_new applies. A slow subscription still sheds alone.

Bounding bytes, not just entries

For a long time the subscriber queue was bounded by entry count only. An entry is a whole published batch, which can be up to 16 MiB. A queue of 4096 entries could in principle hold 64 GiB for one stalled reader (#1157).

So the queue is now bounded by bytes as well (#1159). Each subscriber has a semaphore sized in bytes, 128 MiB by default (FELIX_SUB_QUEUE_MAX_BYTES). Fanout takes permits for the batch's payload size with a non-blocking try_acquire, next to the queue slot. If either is full, the overflow policy applies, same as before.

Then I split that bound in two (#1195, #1243). The queue bound covers what's waiting for the feeder. A second bound, FELIX_SUB_EGRESS_MAX_BYTES, covers what's in the writer lanes and the connection writer, 256 MiB per connection. Each subscription gets an equal share of that and waits only on its own share. A shared pool would let one stalled subscription take all of it and stall its neighbours on the connection.

The same pull request bounded the replay ring each shard keeps for resumes, at 64 MiB, and each cache watch's queue, at 64 MiB (#1201). A watch that would go over ends as lagged, and the client re-watches from where it stopped.

There's also an optional broker-wide total across all subscriber queues. It's off by default, because it gives up the property this post is about. Once slow subscribers fill a shared total, fast ones start dropping too.

The regression

Before a release, I compare it against the previous one. After #1243 merged, the single-publisher cells were slower than the release before it. This is one publisher on one connection, sending 256-byte messages one at a time, at fanout 1 and fanout 10.

The gap was small. Against the previous release, fanout 1 was 2.4% slower and fanout 10 was 2.1% slower, at p=0.003 and p=0.03. On that runner it was around 6,000 messages a second, so 2% is about 130 messages a second. It held up on every rerun.

The per-PR perf check hadn't caught it. That check compares a branch against its merge-base and flags a change only past 3%. #1243 was also judged on a different cell, 64 KiB batches at fanout 10, which got faster.

How I found it

Each step of the hunt was a throwaway draft pull request whose only job was to run the perf workflow. The workflow builds the baseline and the candidate and runs both on the same runner, 15 trials each, and compares them with Welch's t-test. Running both sides on one machine removes most of the noise you get on shared CI runners. This kind of difference is invisible in a comparison against numbers stored from last week.

The bisect landed cleanly. The commit just before #1243 matched the previous release, at +0.6% and +0.7%. #1243 itself was at -2.2% and -0.8% (#1289, #1290).

#1243 changed several things, so I tried them one at a time in the same kind of throwaway PR. A lock-free version of the release I was suspicious of moved nothing, +0.1% and -0.2% against main.

What did move it was where the feeder gave a batch's bytes back to the subscriber's queue bound.

The fix

In #1243, the feeder released the subscriber's bytes as soon as it took a batch off the queue. That was the point of the split: once the feeder has a batch, it counts against the egress share, not the queue. The release happened inside carry, before the frame was encoded or handed to a lane.

The fix keeps the hold until the frame is in the lanes (#1309):

// Given back at the end of this pass, once the frame is handed to the
// lanes. Giving it back before encoding put the release on the
// delivery path and cost single-publisher throughput.
let mut subscriber = queued.take_subscriber();
held.carry(queued);
// ... encode the frame, coalescing anything else already queued ...
enqueue_lane_frame(&manager, lane_idx, subscription_id, frame, len, at, held).await;
// `subscriber` drops at the end of the pass, and with it the bytes.

If the feeder is about to wait out its flush delay for more events, it drops the hold before waiting, so an idle wait doesn't sit on the subscriber's bound.

Against the previous release, the single-publisher cells came back to -0.4% and -0.0%, neither significant (p=0.45 and p=0.83). Tail latency was level too, with p99 at +0.0% and -1.4%. The 64 KiB fanout-10 cell was 10.8% faster than the previous release (p=0.0004). Most of that came with #1243's split itself. The fix kept it.

Why a one-line move mattered

With one publisher sending one small message at a time, each message runs through a chain of steps. It's appended, fanned out, picked up by the feeder, encoded, handed to a lane, and written. With one publisher and messages this small, there's little else going on to hide a step behind. Anything added to that chain shows up as fewer messages a second.

The release was in the chain. It ran after the feeder took the batch and before the frame could move on. Moved to the end of the pass, the same work runs while the writer is already sending the frame. It still costs the same number of cycles. They just aren't in the line anymore.

That fits the lock-free result. Making the release cheaper didn't help, because the problem wasn't that it was expensive. It was that the message had to wait for it. I didn't profile it down to the exact instructions, and I'd rather say that than make up a cache-line story.

One honest caveat. A later experiment showed that Felix's build is sensitive to code layout: adding a single call that does nothing to a cold startup function moved one benchmark cell by about 2% in either direction. Both the loss and the recovery here are inside that range. The consistent direction across both cells, the lock-free result and the clean bisect point make me believe the release placement was the cause, but I can't fully separate it from layout.

What it costs

The subscriber's queue bound now covers a little more than it did. It holds a batch through the encode and the hand-off to the lane, and under a block lane policy, through any wait for lane room. That's one frame per subscription, held slightly longer.

Releasing at the moment of the write was faster again. But that would put the batch back under the queue bound while it sits in the lanes, which is what #1195's split was meant to remove: a batch being written shouldn't look like the subscriber falling behind. I kept the split.

What we learned

Where a cost sits matters more than how big it is. A semaphore release is cheap. On the path a message has to wait for, even a cheap step shows up in a single-publisher benchmark. Off that path, the same step is free.

Small regressions hide well. A 2% loss sits under the per-PR threshold and looks like noise on any single run. It took a same-runner comparison with enough trials and a significance test to see it at all.

Compare against the last release, not just the last commit. Each PR was checked against its merge-base, and each one looked fine. The release comparison is what showed that one of them had cost 2%.

A pull request can be judged on the wrong cell. #1243 targeted large payloads and was checked there, where it got faster. The cost landed in the small-message path, which nobody was watching for that change.

What we'll do differently

Every release gets an A/B against the previous release before it's tagged. Same runner for both sides, 20 trials, on three runner classes, Welch's t-test per cell. A release has to match the previous one; a significant loss gets bisected and fixed, not written up.

When a change touches the delivery path, it gets checked on the single-publisher cells as well as the cells it was written for.

Trying it

The Docker Compose guide runs a control plane and a broker with docker compose up -d (or podman compose). The byte bounds and overflow policies are described in the broker configuration docs.

To watch isolation directly, task demo:slow-consumer runs one stream with three consumers and stalls one of them partway through. It runs the same scenario under drop_new and block and puts the results side by side.

docker compose up -d