Introducing Felix: one log for streams, caches and counters

Felix is a replicated log that serves streams, caches, counters and consumer groups from one system, over QUIC. You publish to it, subscribe to it, read the newest value for a key from it and add to a counter in it, with one client, one wire protocol and one set of permissions.

I built it because most services I've worked on end up running three systems for what is really one job, and the bugs live in the gaps between them. This post is the introduction I'd give an engineer who has never heard of Felix: what problem it removes, how a publish gets through it, why it is built the way it is, and how to have it running on your laptop in about five minutes.

One log, three readings

Take an order service. It needs a record of what happened, so other services can react and so you can replay it. It needs to answer "what is order 42 right now?" without walking that history. And it needs to hand out work, like sending the receipt, to exactly one worker at a time.

The usual answer is an event log for the first, Redis for the second and a work queue for the third. Each is good at its job. But that is three deployments, three permission models, three ways to fail, and some glue code that writes the same fact to two places. When the second write fails, the log says refunded and the cache says paid. Neither system can tell, because only your code knew they were supposed to agree.

Felix stores one thing: an append-only log, where writes land at the end and nothing already written changes. A stream, a cache and a queue are three ways of reading it. A subscriber is a position that walks forward over every record. A cache key is a marker on the newest record carrying that key. A consumer group is a shared position that only moves when a worker says a record is done.

Because they read the same kind of log, they share everything under it: one storage engine, one recovery path, one replication path and one answer to "is this write safe yet?" Access is granted per tenant and checked at the broker, and that one check covers publishing, subscribing and cache operations. When you get paged, it is about one system.

Counters sit beside the cache. counter_add applies a signed delta and answers with the new sum in the same round trip, which is what you want for rate limits, quotas and live tallies. The sum is rebuilt by folding the log, so it survives a restart and a leader failover. Felix also has an atomic commit that writes an event and the state that goes with it as one record on one shard, so a reader never sees one without the other.

How a publish travels

A cluster runs two kinds of process. Brokers hold the data and serve clients over QUIC. A control plane holds the metadata, places each shard of a stream on a broker, and decides which broker leads it. A publish only touches the brokers, and on its way through one it crosses four steps in a fixed order.

First the broker decodes the frame from the client's QUIC stream and checks the client's token allows publishing there. Then, on a durable stream, it claims the batch's offsets from the log before it waits for anything to reach disk. That fixes the batch's place in the stream the moment its offsets are taken. A sequencer then makes later batches wait behind earlier ones, whether the earlier one succeeds, fails or is cancelled, so delivery order always matches log order.

Next the batch is persisted under the stream's fsync policy, and only then fanned out. Fanout builds one delivery envelope per batch and hands every subscriber a reference to it. The encoded frame is cached in the envelope, so the first subscriber to need it encodes it and everyone else gets a cheap copy of the same bytes. Ten subscribers or five hundred, the publish is encoded once.

That last part is why fanout is what Felix is best at. On three 4-vCPU Azure brokers, with a real identity provider verifying every token, one stream on one broker delivered 256-byte messages to 500 subscribers at 1,004,273 messages a second with zero loss. The publisher's acknowledgement p50 went from 0.18 ms with one subscriber to 0.21 ms with five hundred. The conditions and the full table are on the real-network performance page.

A slow subscriber only slows itself

Every subscriber has its own bounded queue on the broker. When one fills, the default policy, DropNew, drops the arriving record for that subscriber alone. The publisher never waits on it, and neither does anyone else. An operator can choose Block instead, which trades that isolation for never dropping.

In the same Azure setup, 50 subscribers read one stream and 10 of them were slowed to 20 ms per delivery. The publisher's acknowledgement p50 stayed at 0.20 ms, the 40 healthy subscribers each received all 42,000 messages, and the slow ones dropped about 98% of theirs. The loss landed only where the reader fell behind.

Dropping is only safe if you can tell it happened. On a durable stream, every delivered event carries its log offset, so a jump in offsets is exactly a drop. The subscription ends at the first drop with the offset to resume from, and the cluster client resumes there from the log. I wrote up how that path works, and a bug in it, in One slow subscriber shouldn't slow anyone else.

The log on disk

Streams are in memory unless you create them with durable: true and the broker has a storage directory. A durable stream's log is a series of segment files, and those segments are the data. There is no separate write-ahead log in front of them, because the log already is one.

The rule the storage layer is built on is that records are never rewritten. That one property makes recovery simple to reason about: valid bytes end at end of file. Indexes are derived from the segments, never trusted, so a missing or stale index is just rebuilt. Preallocation reserves disk blocks without changing the file's size, so a preallocated tail never looks like data.

Recovery tells two kinds of damage apart. A crash in the middle of an append leaves a partial record at the end of the newest segment. Nothing could have acknowledged a record that was never finished, so it is truncated and the broker starts. A bad checksum with records after it is different: those later records may have been acknowledged, and guessing would silently lose them. So the broker refuses to start and names the segment and position.

Three fsync policies decide when a write counts as safe. OnCommit flushes before the acknowledgement and has no loss window. Periodic, the default, flushes every 250 ms, so a machine crash can lose up to one interval. None leaves it to the operating system, which still survives a process crash because the page cache belongs to the kernel.

OnCommit would be slow if every append paid for its own flush, so Felix uses group commit. An fsync flushes the whole file, so appends that arrive while one flush is running all ride on the next one. On a Mac Studio that took the same code path from 253 durable appends a second at concurrency 1 to 14,387 at concurrency 64. The details are in the durable storage design.

Replication

Each shard has a leader and followers. The control plane grants the leader a time-bounded lease for the shard, and the leader ships its log to the followers. I chose leases and log shipping over running Raft per shard because a broker holds many shards, and a consensus round trip on every publish is a cost the common path should not pay.

A stream or cache picks its consistency. Under Leader, a publish is acknowledged once the leader has it. Under Quorum, it is acknowledged once a majority of the replica set holds it durably, whichever broker the client reached, and one unreachable replica does not stall it. When a leader is lost, a replica that held the log takes over under a new generation, and the generation fences the old leader so it cannot commit under the new one.

The control plane decides every failover, so it has to be reachable for a dead leader to be replaced. It keeps its metadata in Postgres, or in a Raft group embedded in its own instances with no external database. The replication protocol is model-checked in TLA+, and fault-injection tests kill brokers and simulate power loss mid-write. Acknowledged, then forgotten is a good example of what that combination catches.

Growing the protocol without version bumps

Felix's wire protocol is framed over QUIC with TLS 1.3 always on. Each frame has a small header with a flags field, and the flags decide how the payload is laid out. That makes an unknown flag dangerous: a peer that ignored it would confidently misread the body. So a broker rejects an unknown flag bit instead of masking it off.

New features are added as bits that both sides agree on, not as protocol versions. The client offers what it can read on the auth message, which is already the first round trip on every stream, so agreeing costs nothing extra. The broker answers with what it supports, and the client uses only what was advertised.

{"type":"auth","tenant_id":"t1","token":"...","client_flags":25}
{"type":"auth_ok","server_flags":25}

A broker that predates negotiation ignores the field and replies with a plain ok. The only safe reading of that silence is the original three flag bits, so that set is frozen and nothing is ever added to it. Optional fields default to the old behaviour, so an old peer and a new one exchange identical bytes. The result is that a newer client runs against older brokers without whatever they predate, and nobody has to upgrade in lockstep. The full rules are in the wire protocol reference.

Kafka clients and browsers

You don't have to rewrite every client to try Felix. Each broker can run a Kafka listener, and a durable stream then looks like a Kafka topic: partitions are shards and offsets are Felix's own. Kafka producers write to it, idempotent ones included, and consumers that assign their own partitions read from it, so programs built on librdkafka or kcat can be pointed at Felix one at a time.

Consumer groups and transactions are refused on purpose, with an error that says why. That means Kafka Connect, Streams and ksqlDB don't work against it. Point your Kafka client at Felix walks through how a Kafka request becomes a Felix operation and exactly where compatibility stops.

Browsers can't open QUIC connections to a broker, so felix-gateway sits between them. A browser opens a WebSocket, names one scope, such as a room or a board, and signs in. The gateway exchanges that sign-in for a Felix token that reaches only that scope's streams, caches and counters, and relays messages both ways without decoding payloads. It holds no application state, so one binary serves any app built from those pieces.

For everything else there are native clients. The Rust client is the reference, and Python and TypeScript bind to it rather than reimplement it. All three are published as felix-client on crates.io, PyPI and npm.

What people build on it

The best test of "one system" is an application with no database next to it. The projects page collects the ones in the GetFelix organization, each written partly to find where Felix falls short.

felix-canvas is a multiplayer whiteboard. Every edit goes to a durable stream and the canvas is a fold of that log, while cursors go to an in-memory stream because a cursor from 40 ms ago is worthless. Presence is a cache whose entries expire, and a counter hands out sequence numbers. felix-webhook-relay stores each webhook before answering, then delivers it to every endpoint through its own consumer group, with retries, dead letters and replay by offset.

felix-arena is a browser team deathmatch whose simulator commits every tick to a durable stream, so the kill cam is a replay of the log. felix-metro is a live 3D city simulator, and felix-blocks, a voxel builder, is in design; both use the same pieces.

Where it fits

Felix fits best where many processes need the same fast-changing data: live feeds, dashboards, presence, configuration and state pushed to a large number of consumers, and services that need an event, a current value and some work queued, all at once. If you're building something new and can see those needs coming, one system you learn once is a real saving.

It is not a drop-in for systems you already run well. If you need the Kafka ecosystem, Redis data structures beyond a value per key, AMQP routing, or history kept longer than your brokers' disks hold, use the tool built for it. The status page marks every capability as built, partial or planned, and it is the page to trust when another disagrees.

Try it in five minutes

You need Docker, or Podman with the docker-compose provider. The compose file in the repository runs a control plane, one broker with durable storage, and development credentials for a tenant called dev, with a durable stream orders and two caches, config and stats.

curl -fsSLO https://raw.githubusercontent.com/GetFelix/felix/main/compose.yaml
docker compose up

When the broker is ready it prints its connection settings. In a second terminal, subscribe, and in a third, publish:

docker compose run --rm felixctl sub orders
docker compose run --rm felixctl pub orders 'hello'

The subscriber prints hello. The same stream is a log, so you can read it again from the start, and the caches and counters are right there:

docker compose run --rm felixctl sub orders --from earliest --count 10
docker compose run --rm felixctl cache watch config   # leave running: prints every change
docker compose run --rm felixctl cache put config theme 'dark'
docker compose run --rm felixctl counter add stats page-views 1

Every port is bound to 127.0.0.1, and the credentials are for local use only. docker compose down -v throws everything away. The Docker Compose guide shows how to point an installed felixctl at the stack, and the quickstart goes further, into the client libraries.

Where to go next

The docs cover the architecture, the wire protocol, configuration and benchmarks, and Why Felix? makes the case at more length, including when not to use it. The source is on GitHub. The broker, control plane and felixctl are AGPL-3.0, and the wire protocol and client libraries are Apache-2.0.

The rest of this blog goes deeper into single pieces: why Felix caps its QUIC packet size on Linux in Ten parcels, one envelope, how a passing TLA+ check can mean nothing in Our model said everything passed, and how a release is held to no measurable slowdown in How we catch a 1% regression.

docker compose up -d