# Operating hundreds of stateful flink pipelines In part 3 of this 3-part series on rebuilding stateful CDC pipelines, we cover how rebuilds became self-service instead of expert-only work. **URL Source:** https://www.brex.com/journal/operating-hundreds-of-stateful-flink-pipelines/part-3 --- Operating hundreds of stateful flink pipelines Part 3 of a three-part series on how Brex rebuilds and operates stateful CDC pipelines. A correct rebuild is not yet an operable platform In the first two parts of this series, we described how we rebuild stateful CDC pipelines. [Part 1](#) introduced federated snapshots: each application receives a private historical stream while continuing to consume the shared live mutation stream. [Part 2](#) explained how a small keyed-state operator prevents old snapshot rows from overwriting newer updates. Those techniques solve the data problem, but they do not automatically solve the operational problem. A design that works for five carefully managed pipelines may not work for a hundred independently owned ones. The platform had to turn expert procedures into safe, repeatable workflows. Two planes We organize the system into a control plane and a data plane: ![High-level architecture: the control plane reconciles jobs onto the cluster, while the data plane moves changes from PostgreSQL through CDC and Kafka into Flink and out to its sinks.](https://brand.brex.com/transform/af614d61-2cf0-4abd-83de-1c7f385f67b3/CDC-blog_inline-7) *High-level architecture: the control plane reconciles jobs onto the cluster, while the data plane moves changes from PostgreSQL through CDC and Kafka into Flink and out to its sinks.* The **data plane** carries changes through the system. The **control plane** owns the lifecycle of the applications running on that path. Keeping those responsibilities separate lets the data path remain focused on processing while the control plane absorbs operational complexity. Declarative application deployment We run Flink on Kubernetes using the open-source [Flink Kubernetes Operator](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.15/docs/concepts/architecture/). ![The operator turns a FlinkDeployment resource into a running cluster: a JobManager and TaskManagers, with configuration and high-availability ConfigMaps.](https://brand.brex.com/transform/4b0c8087-b171-4f05-914b-fc084adf7817/CDC-blog_inline-6) *The operator turns a FlinkDeployment resource into a running cluster: a JobManager and TaskManagers, with configuration and high-availability ConfigMaps.* Each application is declared as a `FlinkDeployment` resource. The operator continuously reconciles the running application toward that desired state. A deployment produces a dedicated Flink cluster for the application, with a JobManager coordinating the job and TaskManagers executing its operators. High-availability metadata allows the application to recover across restarts. The important property is not the Kubernetes object itself. It is that the desired state is explicit. An application owner does not need to manually resequence imperative cluster actions. The operator can compare the declared configuration with the running system and converge toward it. That gives the platform a stable foundation for higher-level lifecycle workflows. A self-service lifecycle We expose common operations through an internal command-line tool. Teams can: - deploy a new application - upgrade an existing application - restart from the latest checkpoint - create a savepoint - restore a previous state - suspend and resume processing - reset a job - request a federated snapshot - begin a blue/green rebuild The CLI is intentionally not a thin wrapper around the Kubernetes API. Many of these actions are multi-step procedures with ordering constraints, retries, and failure recovery. A reset followed by a backfill is not one atomic call. A blue/green deployment, for example, may involve a new production job, a snapshot connector, readiness checks, a publication gate, and retirement of the old version. To execute these steps durably, we use Temporal workflows. Why durable workflows matter Infrastructure operations fail in partial and inconvenient ways. A process can crash after creating a connector but before recording its identifier. A deployment can succeed while a readiness check times out. A retry can accidentally duplicate a destructive operation. A human can close a terminal midway through a long-running backfill. A durable workflow assigns an explicit state machine to each operation. For example, a backfill workflow can: 1. verify application ownership 2. validate the requested tables and snapshot configuration 3. provision a private snapshot topic 4. create the temporary Debezium connector 5. wait for connector completion 6. monitor topic consumption 7. verify the Flink job has processed the final records 8. clean up temporary resources 9. record the result Each step can be retried according to its own semantics. Ultimately, the workflow owns the operational knowledge. Application teams simply invoke an intent, such as “backfill this pipeline,” rather than reproducing a runbook. Ownership is part of the platform Self-service does not mean unrestricted access. The control plane enforces which teams own which pipelines and which operations they are allowed to perform. Teams can operate their applications without gaining direct control over unrelated jobs or shared cluster infrastructure. This matters because lifecycle operations affect durable state. Restoring an old savepoint, resetting a job, or beginning a production cutover should be easier than filing a ticket, but it should still be attributable, authorized, and auditable. Moving those checks into the platform also reduces the number of one-off operational paths the infrastructure team must support. Blue/green for stateful jobs A stateful deployment has a complication that a stateless service usually does not: the new version may need hours to reconstruct its working state. Resetting the current job and rebuilding in place is operationally simple, but downstream output remains incomplete until the backfill finishes. For pipelines that cannot tolerate that gap, we use blue/green deployment. - **Blue** is the current production job. - **Green** is the candidate version. Both consume the same live mutation stream. Green also consumes a private federated snapshot and reconstructs its historical state. During this phase, green checkpoints normally but suppresses production publication. It behaves like a running pipeline without becoming a source of truth for downstream consumers. Blue continues serving traffic. Once green is ready, the control plane promotes it for publication. Blue and green may overlap briefly, so downstream consumers must use idempotent or upsert-style semantics. After the cutover, blue is retired. ![Before promotion, Green app sinks to a suppressed output. After promotion, Blue is orphaned.](https://brand.brex.com/transform/746fe526-fe58-4148-9d24-5ccd86e6da2f/CDC-blog_inline-5) *Before promotion, Green app sinks to a suppressed output. After promotion, Blue is orphaned.* Blue/green is not necessary for every application. Some consumers can tolerate an incomplete output window and use a simpler reset-and-rebuild flow. The platform supports both because operational requirements differ across pipelines. Readiness is more than zero lag “Green has caught up” sounds simple, but the term “production promotion” needs a precise definition. A robust decision can include: - the snapshot connector has completed - the snapshot topic is drained - the live mutation topic is at or below an accepted lag threshold - a successful checkpoint includes the final snapshot records - the job is not experiencing sustained backpressure - checkpoint duration and failure rate are healthy - state size is within expected bounds - application-specific output checks pass Today, some of this validation still includes a human reading dashboards, but one of our next steps is to automate more of the promotion decision. That is an important distinction: blue/green removes the long-serving gap, but a safe cutover still requires evidence that the new job is complete and healthy. Testing stateful changes before production Lifecycle automation is useful only if teams can validate changes before operating on production state. We integrated Flink with Brex's ephemeral-sandbox system so a team can fork a job into an isolated environment connected to realistic staging change streams. The sandbox runs the pipeline end-to-end while remaining separate from shared staging jobs. This allows teams to: - inspect the output - exercise stateful behavior - test topology changes - validate serialization changes - rehearse a state migration - destroy the environment when the test is complete A stateful operator may behave correctly on a small fixture but fail under a long-running sequence of updates, deletes, restarts, or checkpoint restores. An isolated environment gives teams a place to observe the whole lifecycle. Two authoring models, one platform contract Teams build pipelines using two main models: **Flink SQL** Flink SQL works well for declarative transformations expressed as source and sink tables plus SQL. It gives teams a familiar model for filtering, joining, aggregating, and materializing change streams. **DataStream API** For custom state, specialized operators, or integrations with external systems, teams use the DataStream API through Kotlin. The two models differ in expressiveness, but they share the same platform conventions. A common set of Kotlin libraries provides: - source adapters - CDC record models - serialization - sink behavior - state-management conventions - deployment integration - metrics and logging That shared contract lets the platform enforce assumptions that matter during rebuilds, such as key stability and replay-safe sink behavior. It also prevents every team from rebuilding the same infrastructure wiring around its business transformation. Observability by default A pipeline can be unhealthy without being down. It may still be running while: - falling behind the mutation stream - accumulating backpressure - taking longer to checkpoint - approaching a disk limit - restarting repeatedly - growing state faster than expected Every application, therefore, receives a standard observability surface. Teams have access to the Flink web UI alongside centralized logs, dashboards and monitor templates that commonly signal on: - job state - restart count - mutation-stream lag - snapshot-stream lag - sustained backpressure - checkpoint duration - checkpoint failures - RocksDB or local disk use - memory pressure For rebuilds, we also care about the relationship between these signals. A snapshot that finishes at the connector but remains heavily backlogged in Kafka is not complete from the application's perspective. A job with zero topic lag but no successful checkpoints after the final snapshot records may not yet be ready for promotion. A backfill that approaches the mutation-marker TTL boundary becomes a correctness risk, not only a performance issue. Shipping these defaults with the platform matters because most application teams should not need to become Flink observability specialists. **Designing for gradual failure** Many stream-processing incidents are not binary. A job may remain in a `RUNNING` state while its effective service quality degrades. Checkpoints may grow from seconds to minutes. Backpressure may slowly propagate upstream. A disk may fill over several hours. Consumer lag may increase only during peak load. The platform, therefore, treats trends as operational signals. The goal is to surface degradation early enough that a team can intervene before the job becomes unrecoverable or the live stream exceeds retention limits. This is also why lifecycle workflows and observability cannot be designed independently. A safe automated action needs reliable signals, and a useful signal should map to an available recovery action. **What scaled and what remains manual** Today, Brex runs more than a hundred stream-processing pipelines across many teams. Together, they maintain several terabytes of derived state and process hundreds of millions of change events per day. Data that once lagged its source by hours is now available within seconds. New pipelines can be backfilled and serving in minutes rather than days, and application teams can do most of that work without routing every operation through the platform team. The remaining gaps are increasingly about automation rather than basic capability. **Faster, less disruptive backfills** We want orchestration to pace snapshots to the current load on the source database, including throttling and retries, so a team can request a rebuild without manually tuning its impact. **Less operational toil** Autoscaling can adjust job parallelism based on the workload, rather than relying on manual tuning. Blue/green validation can become a policy-driven gate rather than a human reading several dashboards before promotion. **Better testing** Operator-level harnesses and realistic synthetic data can make stateful behavior easier to validate before a full sandbox run. **Better debugging** Stream jobs, particularly SQL jobs, can be difficult to reason about after deployment. We want to make intermediate state and operator behavior easier to inspect when the output is wrong. **What made the platform self-service** No single component turned the system into a platform. The result came from combining: - declarative deployment - durable lifecycle workflows - ownership enforcement - isolated validation - shared CDC abstractions - built-in observability - federated snapshots - blue/green cutovers Federated snapshots made rebuild traffic private. The keyed-state merge made the private snapshot safe. Blue/green prevented the reconstruction period from becoming an availability gap. The control plane made those ideas accessible to teams that did not want to become experts in Flink, Kafka, Debezium, Kubernetes, and Temporal just to operate a pipeline. That last step is what allowed the architecture to scale from a technique used by a small infrastructure team into a routine capability used across Brex. **The broader lesson** A platform is not self-service because it exposes more buttons. It is self-service when the safe path is encoded, the dangerous assumptions are enforced, progress is observable, and failures can be resumed without reconstructing a runbook from memory. Stateful stream processing makes that standard especially important. The system carries a durable history, and many operations unfold over hours rather than seconds. By moving lifecycle knowledge into the control plane, we gave application teams autonomy without requiring each team to rediscover the failure modes of stateful infrastructure on its own. That is what made operating hundreds of pipelines possible. ## Related Articles ### [When your infrastructure tool's best interface is a web page, you have an agent problem](https://www.brex.com/journal/terminal-ui-became-an-agent-interface) We built a terminal UI to replace a browser tab. Then we realized we'd built something agents could use too. ### [Rebuilding stateful CDC pipelines without replaying the world](https://www.brex.com/journal/rebuilding-stateful-cdc-pipelines/part-1) In part 1 of this 3-part series on rebuilding stateful CDC pipelines, we explain why shared snapshots stopped scaling and how we made rebuilds private. ### [Long-running agents don't need tools or hosted sandboxes; all they need is bash.](https://www.brex.com/journal/long-running-agents-need-bash) Brex rebuilt its expense audit agent on a bash workbench instead of native tools. Token usage dropped ~80%, and recall climbed at scale. Learn more. ### [CrabTrap: an LLM-as-a-judge HTTP proxy to secure agents in production](https://www.brex.com/journal/building-crabtrap-open-source) Here's what we learned in building and open-sourcing CrabTrap, and why we believe it’s a major step forward in the security of agent harnesses. ### [What if 90% of your prompt is content you can't control?](https://www.brex.com/journal/articles/what-if-you-cant-control-your-prompt) Discover how Brex's audit agent handles messy, real-world context, and why good AI products give judgment a structure to operate inside, not a script to follow. ### [Agent automations for big migrations: How we accelerated framework upgrades by 32x with Cursor](https://www.brex.com/journal/agent-automations-for-big-migrations) How we arrived at this highly efficient system, along with the missteps we made along the way that are worth understanding.