# Rebuilding stateful CDC pipelines without replaying the world In part 1 of this 3-part series on rebuilding stateful CDC pipelines, we explain why shared snapshots stopped scaling and how we made rebuilds private. **URL Source:** https://www.brex.com/journal/rebuilding-stateful-cdc-pipelines/part-1 --- Rebuilding stateful CDC pipelines without replaying the world Part 1 of a three-part series on how Brex rebuilds and operates stateful CDC pipelines. State is the hard part At Brex, a change in a transactional database rarely stays in one place. When a card is issued, an expense is updated, or a payment is posted, that change may need to refresh a search index, recompute a machine-learning feature, update a materialized view, or feed an analytics pipeline. We use Debezium to capture database changes, Kafka to transport them, and Apache Flink to continuously transform those events while maintaining the derived state our applications depend on.. Moving the events is only part of the problem, though. The harder part is maintaining the state built from them. A stateful stream-processing job gradually accumulates indexes, aggregates, entity views, and other derived data to create its “state” at any point in time. Eventually, that state must be rebuilt. A job's logic may change, a bug may corrupt the current state, a pipeline may need to bootstrap from historical data, or a topology change may make an existing savepoint incompatible. Whatever the trigger, rebuilding is not an edge case; it is part of the lifecycle of a stateful pipeline. The obvious approach sounds simple: 1. Take a snapshot of the source database. 2. Replay the snapshot through the pipeline. 3. Continue processing live changes. At a small scale, that can work well. On a larger scale, the operational cost becomes much harder to ignore. Why shared snapshots stopped scaling for us When we first built our change data capture (CDC) platform, we used Debezium's [incremental snapshots](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-incremental-snapshots). Incremental snapshots are a useful mechanism. They read a table in chunks while the connector continues consuming the live transaction log. Snapshot rows and live mutations are reconciled so the stream converges on a consistent view without pausing database writes or taking the pipeline offline. The approach was correct, but two properties became increasingly painful as the platform grew. Rebuilds were too slow Snapshotting still requires scanning the source table and coordinating that scan with the live transaction log. For our largest tables, a rebuild could take days. That may be acceptable when a pipeline is created once and then left alone. It is much harder to accept when rebuilds become routine: after logic changes, schema changes, state migrations, or operational incidents. A rebuild mechanism that takes days discourages teams from evolving their pipelines. Rebuilds were not isolated The larger problem was not only how long a snapshot took, but also where the snapshot rows went. Historical rows were published into the same shared CDC topic used for live mutations. Every downstream job consuming that topic, therefore, had to read the snapshot replay, even if only one job needed to rebuild. Imagine three pipelines consuming changes for the same source table: - a search pipeline updating OpenSearch - a fraud pipeline computing real-time features - an analytics pipeline maintaining aggregate tables The analytics pipeline changes its schema and needs a full rebuild. With the shared snapshot design, the historical rows are replayed into the common CDC topic. The search and fraud pipelines do not need to be rebuilt, but they still have to consume and process the replay before returning to the live tail of the stream. One team's maintenance operation becomes everyone's extra load, lag, and operational risk. Correctness was no longer sufficient. Rebuilds also needed a predictable duration and per-pipeline isolation. Separate history from live changes The key insight was to stop treating historical data and live mutations as one physical stream. Live mutations are shared because they represent the canonical sequence of changes from the database. Historical snapshots are different: they exist to initialize or rebuild one consumer's state. That suggested a different architecture: - keep live `INSERT`, `UPDATE`, and `DELETE` events on the shared mutation topic - give each Flink application its own snapshot topic - merge the two inputs inside the application We call this approach **federated snapshots** because each application owns an independent historical stream while continuing to consume the shared live stream. The rest of the job still sees one logical stream. The difference is that the rebuild traffic is private to the job that requested it. How a federated snapshot works ![Federated snapshot data flow: a job's composite source joins the shared change topic with its own snapshot topic by primary key and emits the result downstream.](https://brand.brex.com/transform/10eee32b-c15b-48b8-be1b-2409fdd98705/CDC-blog_inline-4) *Federated snapshot data flow: a job's composite source joins the shared change topic with its own snapshot topic by primary key and emits the result downstream* Each Flink application reads from two sources. The first is the shared mutation topic. It carries the live CDC stream: inserts, updates, and deletes from the database. The second is the application's snapshot topic. It carries historical Debezium `READ` events for the table or tables being rebuilt. A custom composite source reconciles those inputs and emits one stream downstream. When a team requests a backfill, the platform provisions a temporary Debezium connector for that application. The connector publishes snapshot rows only to the application's private topic. When the snapshot finishes, the second input becomes idle, and the application continues consuming the live stream as usual. The private topic changes the economics of a rebuild in two important ways. **Rebuild traffic is isolated** A backfill for the analytics job no longer appears in the topics consumed by search or fraud. Those jobs continue processing only live mutations. The team requesting the rebuild pays the cost; unrelated consumers do not. **Snapshot capacity can be tuned independently** Because each application has its own snapshot stream, its partitioning and processing capacity can be adjusted without changing the shared mutation path. A large pipeline can allocate more parallelism to its rebuild without forcing every other consumer to scale up or absorb the same replay. This is the architectural shift that made routine rebuilds practical for us: historical data became a per-consumer concern rather than a shared event-stream concern. The merge is where correctness lives Separating the streams solves the isolation problem, but it creates a new correctness problem. Snapshot rows are historical by definition. Live mutations continue arriving while the snapshot is being read. If a historical row arrives after a newer update, the snapshot must not overwrite the live value. For example: 1. The snapshot reads account `A` with status `pending`. 2. A live mutation changes account `A` to `approved`. 3. The snapshot row arrives at the Flink job after that mutation. The correct output is `approved`, regardless of which Kafka input delivers its record first. Our composite source solves this with keyed state and a time-to-live. At a high level: 1. Events are partitioned by primary key. 2. When a live mutation arrives, the operator records a marker for that key. 3. When a snapshot READ arrives, the operator checks for the marker. 4. If a marker exists, the snapshot row is stale and is discarded. 5. If no marker exists, the snapshot row is emitted. Deletes follow the same rule. A delete mutation records the key and prevents an older snapshot row from resurrecting the deleted entity. The central invariant is simple:_ A snapshot row may be emitted only if the operator has not already observed a newer live mutation for that key._ The subtle part is retention. The mutation marker must remain in state until the corresponding snapshot row has had enough time to arrive. If the marker expires too early, a stale historical row can slip through. That is why the operator's TTL must comfortably exceed the longest expected snapshot duration. The exact reasoning, failure modes, and timing cases deserve their own treatment. We cover them in [Part 2: Making CDC Snapshot Merges Correct](https://www.brex.com/journal/making-cdc-snapshot-merges-correct/part-2). Rebuilding without serving incomplete state Federated snapshots let a job rebuild independently, but a rebuilt job still needs time to reconstruct its state. The simplest approach is to reset the job, begin the federated snapshot, and let it publish as it rebuilds. That requires little coordination, but its output is incomplete until the backfill finishes. For some pipelines, that temporary gap is acceptable. For latency-sensitive consumers, it is not. For those jobs, we use a blue/green deployment model. - **Blue** is the current production job. - **Green** is the new version rebuilding in parallel. Both consume the same live mutation stream. Green also consumes its private snapshot stream and reconstructs its state while suppressing production output. Blue continues serving downstream consumers. Once green has completed the snapshot and caught up to the live stream, we promote it. There is a brief overlap during which both versions may publish, so this workflow depends on idempotent or upsert-style downstream consumption. We keep that overlap short and then retire blue. The result is not a magical duplicate-free handoff. It is a deliberate tradeoff: - avoid a multi-hour availability gap - tolerate a short at-least-once overlap - rely on idempotent sinks to preserve the final state We cover the surrounding deployment, validation, and operations model in [Part 3: Operating Hundreds of Stateful Flink Pipelines](https://www.brex.com/journal/operating-hundreds-of-stateful-flink-pipelines/part-3). What this changed for teams Federated snapshots changed rebuilds from a platform-wide event into an application-level operation. A team can now: - request a backfill for one pipeline - scale that snapshot independently - rebuild while continuing to process live changes - validate a new version alongside the existing job - promote it without making unrelated consumers replay history The architecture does introduce costs. Each application needs snapshot-topic infrastructure, and the merge operator retains additional keyed state while a backfill is running. That increases Kafka storage and temporarily enlarges Flink checkpoints and savepoints. For us, those costs were easier to manage than the shared replay tax. Today, more than a hundred stream-processing pipelines at Brex maintain several terabytes of derived state and process hundreds of millions of changes each day. Rebuilds that once required careful, centralized coordination have become a normal part of operating the platform. The broader lesson The most important lesson was not specific to Flink or Debezium. Historical data and live data may describe the same entities, but they serve different operational purposes. Treating them as one shared stream couples every consumer to every rebuild. Once we separated them, the system became easier to scale technically and organizationally. The live mutation stream remained shared. Historical rebuild traffic became private. A small keyed-state operator restored the logical view. Blue/green deployments hid the reconstruction period from downstream systems. That combination turned state rebuilding from an operational fire drill into a routine workflow. ## Related Articles ### [Operating hundreds of stateful flink pipelines](https://www.brex.com/journal/operating-hundreds-of-stateful-flink-pipelines/part-3) In part 3 of this 3-part series on rebuilding stateful CDC pipelines, we cover how rebuilds became self-service instead of expert-only work. ### [The future of financial reporting is not a chart](https://www.brex.com/journal/articles/the-future-of-financial-reporting-is-not-a-chart) Meet Brex Spaces, an AI-powered workspace for finance teams that goes beyond charts to deliver powerful insights, instantly. ### [Agent automations for big migrations: How we accelerated framework upgrades by 32x with Cursor](https://www.brex.com/journal/agent-automations-for-big-migrations) How we arrived at this highly efficient system, along with the missteps we made along the way that are worth understanding. ### [How we built an agent that turns customer feedback into shipped fixes](https://www.brex.com/journal/agent-that-turns-customer-feedback-into-shipped-fixes) Brex built an agent workflow that reads customer feedback across tools, finds quality of life improvement opportunities, and fixes them. Here's how it works. ### [How we built an AI oncall engineer at Brex](https://www.brex.com/journal/how-we-built-an-ai-oncall-engineer) We encoded our oncall playbook into an agent. Here's what happened. ### [Long-running agents don't need tools or hosted sandboxes; all they need is bash.](https://www.brex.com/journal/long-running-agents-need-bash) Brex rebuilt its expense audit agent on a bash workbench instead of native tools. Token usage dropped ~80%, and recall climbed at scale. Learn more.