Tableflow: Turn Kafka Topics into Iceberg Tables

Tableflow is a feature in Confluent Cloud that streamlines the conversion of Apache Kafka topics into Apache Iceberg or Delta Lake tables, allowing analytics engines to query streaming data without the need for manual ETL pipelines. By using Confluent's Kora storage layer, Tableflow transforms Kafka data into structured Parquet files and generates the necessary metadata for Iceberg or Delta Lake, subsequently publishing these tables to a REST catalog. This process eliminates the need for custom data pipelines, addressing the common issue of operational data residing in Kafka while analysts require data in data lakes or warehouses. Tableflow automatically handles schematization, type conversions, schema evolution, and catalog publishing, ensuring seamless integration with analytics engines like Snowflake, Databricks, AWS Athena, Amazon Redshift, Trino, and BigQuery. It supports schema evolution through Confluent Schema Registry, which governs compatibility and automatically applies compatible schema changes during the materialization process. This enables organizations to maintain analytics-ready data continuously in sync with their real-time Kafka data, using Amazon S3 as the storage layer and optionally integrating with AWS Glue or other Iceberg-compatible catalogs for SQL-based access.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.