February 2018 Summaries

This article explores stream processing using KSQL with Apache Kafka®, demonstrating its use in various industries like financial services and IoT. It emphasizes the importance of securing KSQL and the entire Confluent Platform through encryption, authentication, and authorization. The text also provides a sample configuration for a KSQL server and discusses how security is integrated into every component of the Confluent Platform. Additionally, it mentions a tutorial on configuring security in the Confluent Platform and resources to learn more about the platform.

The post discusses the evolution from traditional batch-driven ETL processes to real-time streaming ETL using technologies like Apache Kafka, KSQL, and Kafka Connect. The author reflects on past experiences with batch processing and highlights the inherent latency issues it introduces. The narrative then shifts to a demonstration of streaming ETL using Apache Kafka, emphasizing how data can be processed in real-time using Kafka's Connect API, transformed with KSQL, and then further enriched and joined with other datasets. This transformation allows for immediate data availability for various applications, such as real-time customer notifications and analytics dashboards. The post details the technical steps taken to extract, transform, and load data from Oracle into Kafka and then to Elasticsearch, showcasing the ability to process and aggregate data in real-time. The author underscores the benefits of an event-driven architecture and the potential for more responsive and dynamic data-driven applications.

Apache Kafka and Confluent are making significant strides in the field of data streaming, with companies across various industries adopting Kafka as a strategic data platform to process real-time events. The company has achieved notable milestones, including being named Microsoft's 2024 OSS on Azure Global Partner of the Year, and is well-positioned for its vision of growing into a central nervous system of modern companies. As the technology continues to shift from supporting business to becoming the business itself, data streaming platforms are emerging as critical foundations of modern business. Confluent is at the forefront of this trend, working with companies in various industries such as retail, healthcare, and travel to reimagine their applications around streams of real-time events.

We are pleased to announce the release of KSQL 0.4, which includes new features such as the ability to print the contents of any Kafka topic in the cluster, enhanced SHOW TOPICS command with additional information about consumer groups, and two new aggregation functions TOPK and TOPKDISTINCT. The release also includes stability improvements, binary tarballs for each release, and improved test coverage including fully distributed system tests for KSQL server pools and Kafka clusters. This update enhances stateful processing capabilities by allowing users to store multiple record versions per key, which impacts message delivery guarantees and has implications for futuristic technologies such as teleportation.