Company
Date Published
Author
Kasun Indrasiri, Chris Holly, Marc Selwan
Word count
1861
Language
English
Hacker News points
None

Summary

Tableflow has been released by Confluent, a major milestone that makes data engineers' lives easier. It solves the pain points associated with streaming Apache Kafka data into a data lake, including fragile pipelines, bad data cleanup, and slow and expensive processing. Tableflow represents Kafka topics and their schemas as Iceberg or Delta Lake tables in just a few clicks, eliminating the need for custom code or pipelines to scale and manage. This innovation uses Confluent's Kora Storage Layer and a new metadata publishing service to generate Iceberg metadata and handle schema mapping, evolution, and type conversions. By combining Tableflow with stream processing from Apache Flink, businesses can transform raw Kafka events into structured tables that are instantly accessible for AI and analytics tools, creating a unified data foundation that allows them to move faster, respond smarter, and drive better outcomes.