December 2019 Summaries

This is a celebratory message from Confluent, marking its 1,000th employee milestone and reflecting on its rapid growth and progress over the past five years. The company has come a long way since its humble beginnings as a tiny office with a clear vision to build an infrastructure platform for event streaming. Confluent's success is attributed to its people, products, and community, which have enabled it to scale its business while innovating and adding value to customers. With 60% of the Fortune 100 relying on Apache Kafka and Confluent Cloud, the company is poised to continue growing and making a significant impact in the event streaming space. The milestone is also seen as an inflection point for data streaming platforms, with Confluent being named Microsoft's 2024 OSS on Azure Global Partner of the Year.

The text discusses ksqlDB, a stream processing engine that can be used for building data pipelines and applications. It covers the basics of ksqlDB, including its ability to process data from Kafka topics, create streams and tables, and perform aggregations. The article also explores how ksqlDB can be used with connectors to read data from external sources such as Twitter, and how it can push data directly out to datastores like relational databases and graph databases like Neo4j. Additionally, the text touches on the concept of stateful aggregation and how ksqlDB can provide real-time access to aggregate values. The article concludes by highlighting the value of using ksqlDB for building stream processing pipelines and applications that require real-time data processing and analytics.

Confluent Platform 5.3 introduces CP-Ansible, open-source Ansible playbooks designed to simplify the deployment and configuration of Apache Kafka and the Confluent Platform by leveraging Ansible's agentless, YAML-based automation capabilities. CP-Ansible aims to enhance the setup process by applying best practices for infrastructure management and offering customization options through variable overrides in the playbooks. This approach facilitates seamless security protocol deployment and allows for scalability across multiple hosts, ensuring correct configurations for communication between components like brokers, Apache ZooKeeper, and Confluent Schema Registry. The playbooks also support the SASL PLAIN security protocol, simplifying security deployments across clusters. Users can validate their installation via the Confluent Control Center interface, which provides an overview of cluster statuses and statistics. The article encourages community contributions to CP-Ansible and highlights the availability of commercial support from Confluent. The blog also notes the upcoming enhancements to CP-Ansible and introduces new features in Confluent Platform 7.7, including OAuth support and integration with Apache Flink and Amazon OpenSearch.

The sticky partitioning strategy in Apache Kafka 2.4 aims to improve performance by sending all records to one batch and potentially filling it earlier, reducing latency when producing keyless messages. The new partitioner, DefaultPartitioner, implements this feature, which can lead to significant reductions in latency compared to the original default partitioning strategy. The sticky partitioner is particularly beneficial in scenarios with high throughput, low latency requirements, or a large number of partitions. By sending fewer but larger batches, the producer experiences improved performance, including reduced CPU usage. This improvement becomes more pronounced as the number of partitions increases.

The text provides an in-depth guide on testing Kafka Streams applications using TopologyTestDriver, particularly emphasizing the advantages of using this method over older approaches like EmbeddedSingleNodeKafkaCluster. The guide highlights the introduction of TestInputTopic and TestOutputTopic classes in Kafka version 2.4.0, which simplify the testing process by reducing the complexity of producing and consuming records and validating them with assertion frameworks like AssertJ. The tutorial includes various examples of testing scenarios, such as validating output records' keys and values, handling timestamps, and managing record headers, providing sample code and explanations for each. Additionally, the text mentions that users can migrate existing tests with simple modifications and offers resources for further exploration, including GitHub repositories and Apache Maven packages. Overall, the post aims to streamline the testing of Kafka Streams logic by offering practical solutions and encouraging the use of efficient testing practices.

Apache Kafka 2.4.0 has been released with several key new features and improvements, including support for reading from follower replicas in multi-datacenter deployments, incremental cooperative rebalancing for consumer groups, and a new partitioner that helps improve message batching and reduce latency. The release also includes enhancements to the Kafka remote procedure call (RPC) protocol, such as optional tagged fields and more efficient serialization of variable-length objects. Additionally, this version introduces a new Java authorizer API with improved security features and better support for asynchronous ACL updates. Other notable improvements include support for default broker configurations during topic creation, retained completed fetch data when partitions are paused, and out-of-the-box authentication and authorization for Connect workers. The release also includes several other feature enhancements and bug fixes, making it a significant update for Kafka users.

The SMT is used to replicate Kafka topics between separate Schema Registry clusters without overwriting pre-existing schemas, ensuring schema compatibility and consistency across environments. The transform includes a revision of the schema ID alongside the topic's replication configuration, allowing for selective transfer of schemas between clusters. This enables seamless integration of Confluent Platform with multi-datacenter environments, where Schema Registries do not require an active-passive configuration and can be shared across multiple clusters without potential schema ID collisions.

The text discusses the use of asyncio and coroutines in Python, specifically in the context of asynchronous web frameworks. It explains how async/await syntax allows for cooperative multitasking, which can improve performance by allowing the event loop to manage multiple coroutines simultaneously. The text then focuses on building an example producer class using the Confluent Kafka library, which is designed to work with asyncio and coroutines. The producer class uses a callback-based approach to communicate with the Kafka broker, but also provides an async/await interface for users who prefer a more Pythonic API. The example demonstrates how to use this producer class in a FastAPI web application, measuring its performance under load testing. Additionally, the text touches on other topics such as consuming messages from Kafka, using the AdminClient, and adding queue support to Kafka.

Deutsche Bahn (DB) has implemented the RI-Plattform, a sophisticated passenger information system, to ensure consistent and accurate information for its 5.7 million daily rail passengers. This system processes around 180 million events daily by utilizing Apache Kafka, Kafka Streams, and Confluent Platform, alongside technologies like Apache Cassandra and Kubernetes on AWS. Initially exploring Apache Storm, DB found Kafka Streams to simplify their environment, enhance development, and improve processing times significantly. The RI-Plattform, supported by 110 developers in 13 scrum teams, comprises 100 microservices communicating through Confluent Platform, achieving 99.9% availability with minimal downtime in its first year. Currently focused on intercity travel, DB plans to expand to intracity travel, increasing the system's load twentyfold, and is considering further enhancements with ksqlDB and Confluent Cloud. The pilot phase covers 80 stations in Germany, and the system's real-time data handling, such as sensor-triggered train announcements, underscores the importance of low-latency event streaming, with future improvements planned through Kafka's Static Membership protocol.