The data came back

Delete a durable stream in Felix, create a new one with the same name, and publish nothing. Subscribe from the start. Before incarnations, you could get the old stream's records back.

The broker named each shard's directory after the stream's name: tenant, namespace, stream, shard. Deleting the stream unregistered it but left the directory. Creating it again opened the same directory, and recovery did what recovery does. It found valid segments and served them.

Fixing that was a storage change. Rolling the fix out was the harder part, because the control plane is a Raft group, and members on two releases have to agree on every entry.

What we found

The first bug was on one broker (#1208). A recreated stream got everything the old one had left on disk: its records, its offsets, so it didn't start at 0, its idempotent producer state, its consumer-group positions and its dead letters. A producer on the new stream could be told its first send was a duplicate. Caches and counters had the same problem, for the same reason (#1209).

A cluster added two more. Deleting a stream dropped its shard assignments without recording a change, so brokers kept the old ones. The new stream's first assignment came in at generation 0, and a broker that held the old shard ignores any assignment that isn't above the generation it has. It simply didn't take the new stream.

The third was in replication. A follower took records for a shard by name and generation. Nothing told it which stream they belonged to, so it could write the new stream's records into the deleted stream's log, and the leader counted them toward a Quorum acknowledgement.

Why it matters

The obvious harm is cross-talk. Two unrelated streams that happen to share a name are the same data as far as the disk is concerned. Someone deletes orders in a test namespace and recreates it for a different service. That service's consumers then read the old one's records, and nothing logs an error. The offsets don't even start at 0.

The cluster case fails the other way. The recreated stream exists in the control plane, but a broker that held the old shard won't open it. The replication case is quieter still. A record acknowledged under Quorum could be sitting in a log that the follower deletes as soon as its catalog catches up.

The rollout risk was worse than any of these. Get it wrong, and members of the control plane hold different metadata for the same stream for good.

How we found it

Nobody hit this in a running cluster. I found #1208 while reviewing an unrelated change, by reading the delete path and noticing it never touched the disk. A regression test, stream_recreate.rs, confirmed it: delete, recreate, and the old records are there.

Each fix turned up the next problem. Fixing the directories exposed the generation-0 assignment and the replication gap. Fixing those for streams left caches, which became #1287. The review of #1287 found the divergence described below.

The fix

The idea is to stop treating the name as the identity. Each creation of a name gets an incarnation, and everything that holds data for a stream holds it per incarnation (#1210).

The create proposes the control plane's clock in microseconds. The store decides the final value inside the same write, so every replica decides alike. It is the larger of the proposal and the name's tombstone plus one, so a name's incarnations only go up. I used the clock rather than the change sequence because a restore rewinds the sequence. A create after a restore should still land above everything handed out before it.

On the broker, each incarnation has its own directories. Incarnation 0 keeps the name a shard always had, so nothing moves on disk during an upgrade. Anything else gets the incarnation as a suffix:

pub fn incarnation_dir_name(shard: &ShardKey, incarnation: u64) -> String {
    let name = shard_dir_name(shard);
    if incarnation == 0 {
        name
    } else {
        format!("{name}-{incarnation:016x}")
    }
}

A broker never moves a name to a lower incarnation, or to one the control plane has deleted. It deletes data only on the control plane's word or an operator's. A delete writes a marker first, moves the directories to .deleted/, then clears the marker, so a crash in the middle is finished at the next start.

That word has to reach a broker that was down through the delete. So every delete records a tombstone: one small row per name that says “every incarnation up to N is deleted”. The catalog snapshot serves them, and a broker applies them before it registers any stream. A broker that slept through a delete and a recreate still removes the old data when it comes back.

Shard ownership now compares incarnation first, then generation (#1245). An incarnated stream's first generation is the change sequence, so on one timeline it already lands above the old stream's. After a restore it might not, and the incarnation decides.

Replication requests name the incarnation in a small Incarnated envelope. A follower holding a different one answers StaleRoute, and the leader tries again later. Leaders also keep follower positions and quorum marks per incarnation. A recreate at the same generation can't inherit the old stream's progress.

The rollout problem

Incarnations change what the control plane stores, so they ship as a fleet feature. Brokers report the features they implement, and an operator finalizes one when the whole fleet supports it. Finalizing is one-way. The control plane refuses it while any serving broker lacks the feature, and refuses an older broker's registration after it.

The Raft side has its own gate. Each member runs at a metadata level, and the group runs at its lowest member's level. v0.6.0-preview.7 is at level 8. Stream incarnations need level 10, and cache incarnations need 14. An upgraded member refuses to finalize stream_incarnations with a 409 until every member is at 10.

The review of #1287 found the hole in that gate. A preview.7 member doesn't know stream_incarnations. It reads its finalize as an ordinary level-2 fleet feature finalize, and proposes it. So the finalize can commit while that member is still in the group.

Once it had, a create proposed by the old member carried no incarnation, because level 8 has never heard of one. The old member stored the stream at 0. The upgraded members applied “tombstone plus one” and stored it at 1. Same entry, same log position, two answers. Raft doesn't notice, because it replicates entries, not results, so the members would stay split for good.

The fix for the rollout

The rule we settled on is that apply decides only from fields a new proposer writes. Whether a member has applied the finalize, or what build it runs, must not change what an entry means.

Every create and delete command gets an incarnations flag at level 14. A proposer sets it only when every member is at 14. A flagged entry is decided from its own fields. An unflagged one is decided the way the older levels decide it. For streams, that means a create that proposes 0 stores 0, finalized or not, and a delete of a stream at 0 writes no tombstone. That's what preview.7 stores too.

pub(super) async fn incarnation_for(
    store: &InMemoryStore,
    nodes: &super::NodeState,
    key: &StreamKey,
    proposed: u64,
    decide: Decide,
) -> u64 {
    // An entry's 0 proposal stays 0: a member before incarnations applies
    // every create that way, even after a finalize it accepted unknowingly.
    if !incarnations_finalized(nodes) || (decide != Decide::Here && proposed == 0) {
        return 0;
    }
    let deleted = store.stream_tombstones.read().await.get(key).copied();
    crate::model::next_incarnation(proposed, deleted)
}

The test that holds this, stream_entries_from_a_level_8_proposer_store_what_level_8_stores, writes entries the way a level-8 proposer writes them and checks the result matches level 8's state. It fails with either rule reverted. Caches have the same test against level 13.

The fix had a price. Levels 10 to 13 existed only on main, never in a tagged release. Builds at those levels gave a 0-proposal “tombstone plus one” after the finalize. A cluster that ran one of those builds and logged such an entry now replays it differently, and those members have to be rebuilt from a snapshot. The other option was to keep the unreleased behaviour and let preview.7 clusters diverge. We protected the release people can actually run.

What it costs

Tombstones are kept forever: one small row per deleted name, overwritten by the next delete. Nothing can tell when every broker has seen a delete, so nothing can safely drop one.

Before the finalize nothing changes. A deleted stream's data stays on the brokers, and a recreate reopens it, as in earlier releases. That data goes the next time the name is deleted after the finalize.

While a member is below level 14, a cache create through an upgraded member is refused with 409. The first flagged entry raises the group's level floor to 14, so a member below it is refused at join and held by the leader. A held voter costs fault tolerance until it's upgraded. With Postgres, an instance from before incarnations can't create streams once the feature is finalized, so every instance has to be upgraded first.

What we learned

A name is not an identity. The directory said orders, and so did the new stream, and the disk had no way to tell them apart. Anything that can be deleted and created again needs something that never repeats.

Every replicated state machine change is also an upgrade problem. The first version of the feature was correct on a group of identical members. It was wrong on a group halfway through a roll, which is the only kind of group that ever applies the switch-on. The gate on the finalize looked like protection, but an old member doesn't know the gate exists.

Check claims about release history against the tags. The first version of the rollout fix assumed that only builds that never shipped could hit the divergence. Looking at the tag showed preview.7 at level 8, which was exactly the release that could. git show v0.6.0-preview.7:<path> answers that kind of question in one line. Memory, or a confident sentence in a PR description, doesn't.

What we'll do differently

Every new field or command that changes what Raft apply computes gets a mixed-level test before it merges. The test writes entries as the oldest released level writes them, and checks the result is identical.

Apply reads only the entry and replicated state. A rule that depends on whether this member has seen a finalize, or on its own build, is a bug even if every member happens to agree today.

When a fix forces a choice between released and unreleased builds, the released one wins. The cost goes in the changelog's upgrade notes.

Trying it

The Docker Compose guide runs a control plane and a broker with docker compose up -d (or podman compose). The upgrades page has the runbook for finalizing stream_incarnations and cache_incarnations, and docs/metadata-raft-design.md lists what each metadata level adds.

docker compose up -d