October 2016 Summaries

Interactive Queries is a new feature in Apache Kafka that enhances stream processing by allowing developers to query the state of a streaming application directly, acting as an embedded database within the stream processing layer. This feature eliminates the need for external databases to materialize state, providing high availability and fault tolerance by leveraging Kafka’s Streams API. Inspired by the concept of materialized views in traditional databases, Interactive Queries simplifies the architecture by integrating processing and storage, making stream processing more accessible as a mainstream application development model. This approach is exemplified in real-time applications, such as financial risk management, where maintaining and querying state efficiently is crucial. By using embedded databases like RocksDB, developers can avoid the inefficiencies of maintaining separate clusters, as seen in the transition from Lambda to Kappa architecture. While the Interactive Queries are read-only to maintain state consistency, they facilitate real-time data consumption for applications like dashboards and microservices, without the overhead of external storage. The implementation of Interactive Queries supports greater flexibility and performance, with the ability to interactively access streaming data, thus bridging the gap between data processing and interactive analytics.

Kafka and the Confluent platform are increasingly becoming central elements in big data architectures across diverse industries, functioning as a nervous system that integrates streaming and batch data for enhanced decision-making. This integration allows companies to process real-time data streams alongside historical data from traditional databases, providing context and enabling applications like fraud detection, real-time reporting, and healthcare analytics. Syncsort partners with Confluent to bridge the gap between these data types, demonstrating the benefits of this approach in various sectors. For instance, fraud detection systems can use real-time transaction data to identify and prevent fraudulent activities promptly, while hotels and healthcare providers can make more informed decisions by analyzing up-to-date data. The collaborative efforts of Syncsort and Confluent illustrate how combining batch and streaming data architectures can transform data processing, making it dynamic and responsive to real-time needs.

Anil Kumar, a Global eCommerce Engineer at Walmart Labs, discusses the evolution and impact of data processing systems at Walmart, highlighting the adoption of Kafka to manage decentralized, autonomous services across multiple team operations. In 2014, Walmart began redesigning its data processing architecture to handle various data characteristics from products, offers, pricing, inventory, and logistics, aiming to improve product listing efficiency and business growth. Kafka's integration has supported the rapid onboarding of sellers and faster product listings, serving as the backbone for a Near Real Time Search Index and processing billions of updates daily, primarily driven by pricing and inventory adjustments. This shift has facilitated agile development and increased operational efficiency through decentralized data management while introducing challenges such as managing service topologies and schema management. The future focus includes increasing internal awareness of Kafka, exploring new streaming technologies like Kafka Streams and Apache Flink, and contributing to the Kafka open-source community.

The Apache Kafka community has released version 0.10.1.0, which includes support for time-based search, replication quotas, improved log compaction, interactive queries, consumer stabilization, and secure quotas. This release was a community effort involving 115 contributors and addresses various bug fixes and improvements. The new features enable finer-grained log retention, predictable performance, and better protection in secure environments. Users can download the latest version from the official website and stay tuned for upcoming releases of Confluent's Enterprise version of Kafka.

The article explores the integration of Oracle databases with Apache Kafka and Elasticsearch using Confluent's Kafka Connect platform. It illustrates the use of Oracle GoldenGate, a real-time data replication tool, to stream changes from an Oracle database into Kafka, and subsequently into Elasticsearch, leveraging the flexibility and power of Kafka Connect. The article details the configuration steps for setting up Oracle GoldenGate and the necessary connectors, emphasizing the advantages of Oracle GoldenGate over JDBC in terms of latency, resource efficiency, and scalability. It also covers the process of creating dynamic templates in Elasticsearch to manage data types and enhance data usability for analytics in Kibana. Additionally, the article guides on monitoring the data pipeline using JMX metrics and testing the setup with tools like Swingbench, showcasing how data can be efficiently streamed and analyzed using this robust integration.

The Apache Kafka community is working on the release of version 0.10.1.0, with contributions from members like Jason Gustafson and Derrick Or. Community discussions are ongoing, focusing on topics such as streaming data platforms and the debate between tabs and spaces in coding. Confluent has released a 6-part online talk series about Apache Kafka, while version 3.8.0 of the software was recently launched with new features and improvements.