Enable “Shift-Left” with Apache Kafka and Iceberg

The blog post explores utilizing the Apache Iceberg table format within DeltaStream to enhance data streaming processes. By integrating data from Apache Kafka topics and leveraging DeltaStream's capabilities, it demonstrates how to read, filter, and enrich data before writing it to Iceberg tables for querying. The process involves setting up a demo Kafka cluster, defining DeltaStream objects for data transformation, and creating Iceberg tables for specific scenarios such as "Pageviews per city per minute" and "Pageviews per user per hour." This approach exemplifies a "shift-left" strategy by moving enrichment and filtering to the streaming architecture, reducing costs and latency associated with batch processing, and enabling real-time data querying. The blog highlights the versatility of DeltaStream in allowing users to query Iceberg tables directly and the potential for further data enhancements without relying on additional tools.