April 2015 Summaries

I observed several trends at ApacheCon 2015, including a strong presence of Apache Kafka, with talks on its scalability, architecture, and integration with other frameworks such as Confluent Platform, Pulsar, and Apache NiFi. I also saw interest in emerging technologies like real-time analytics, stream processing, and data flow management systems. The Confluent Platform was showcased, including Schema Registry and Confluent's efforts to improve server utilization through resource allocation and isolation techniques. Additionally, the event highlighted collaborations between Confluent and other companies, such as Amazon Web Services, to simplify cloud services and data management. Overall, the conference demonstrated a growing interest in real-time stream processing technologies.

Confluent has released an open-source tool called Bottled Water, which allows users to transform their PostgreSQL database into a stream of structured Kafka events. This enables change data capture for Postgres, allowing users to unlock the value in their existing data and build systems that can handle large amounts of data changes. The tool uses logical decoding, a feature introduced by PostgreSQL 9.4, to extract row-level change events from the database's write-ahead log. These events are then encoded in Avro format and sent to Kafka topics, where they can be consumed and processed by downstream systems. Bottled Water is designed to work with Confluent's stream data platform and provides a way to maintain a copy of your database in Kafka, making it easier to experiment with new analysis or data formats, and allowing for gradual migration from one system to another with minimal risk. The tool is currently alpha-quality software and is being released as open source to receive community feedback.

The talk describes open source tools that enable search on streams: Luwak is a Lucene-based library for running many thousands of queries over a single document, with optimizations that make this process efficient. Samza is a stream processing framework based on Kafka, allowing real-time computations to be distributed across a cluster of machines. The speaker discusses how these tools can be combined into an efficient and scalable streaming search engine, and how they can be used to build full-text search engines like those found in Twitter and Google Alerts, with optimizations such as indexing queries and partitioning streams to enable scaling and performance.