Combining CDC Transactional Messages Using Kafka Streams

Thoughtworks developed a Kafka Streams pipeline to replicate changes from a 25-year-old IBM Db2 mainframe database into scalable, domain-oriented microservices without adding load to the source system. A commercial CDC tool emits one Avro message per changed database row, but because messages are randomly partitioned and lack reliable transaction keys, the application first repartitions them by transaction ID, aggregates all records from each transaction in a materialized state store, and releases processing only when the transaction’s final-message indicator confirms completeness. It then converts grouped changes across related tables into business-specific events, such as customer or address creation and modification events, keyed by the relevant domain identifier and sent to separate Kafka topics. The design also handles incomplete or stale transactions through scheduled state-store cleanup and alerts, uses Kafka headers to carry correlation IDs derived from transaction IDs for tracing, supports initial database loads through snapshot events, and relies on downstream consumers to process potentially duplicated events idempotently under an at-least-once delivery model.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.