January 2021 Summaries

SPOUD, a company based in Bern, Switzerland, developed an innovative application called Töggelomat to track table football scores using Apache Kafka, ksqlDB, and Quarkus. This project, initiated during a SPOUD hackathon, served as an exploration of event-driven architectures and real-time data processing, allowing the team to leverage their expertise in Kafka while experimenting with ksqlDB and Quarkus, which they found particularly effective for cloud-based applications due to its reactive nature and efficient performance. The application transforms employees into players with initial scores, randomly generates matches, and updates player rankings based on match outcomes, demonstrating the dynamic nature of event-stream processing. The development process highlighted both the potential and challenges of using event-driven systems compared to traditional database management systems, particularly in terms of maintaining consistency and managing dependencies. The project also underscored the flexibility of using Confluent Cloud for scalable data processing, despite some limitations in service sizing. Overall, the Töggelomat project not only enriched SPOUD's technological repertoire but also provided a platform for potential future enhancements like real-time dashboards and integrations, showcasing the possibilities of modular event-driven applications.

Podium's engineering team has evolved their tech stack from a monolithic Rails app to a microservices architecture running within Kubernetes, incorporating distributed data tools like Apache Airflow and Kafka. This shift has enabled the company to handle millions of conversations simultaneously, with Kafka becoming a foundational piece of infrastructure. By leveraging Kafka's streaming capabilities, Podium was able to upgrade their system performance and decouple their infrastructure, allowing them to "Ship It and See What Happens" while continuing to optimize for feature speed to market. The team has successfully implemented a data pipeline that aggregates messages from various sources and outputs structured data for Elasticsearch, enabling features like conversation search and review management. This approach has enabled Podium to scale well and provide messaging tools to local businesses, modernizing their communication with customers.

This strategic alliance between Confluent and Microsoft enables seamless integration of Confluent Cloud with the Azure platform, accelerating the journey to cloud computing for event streaming and Apache Kafka. The partnership converges the benefits of cloud computing with an open-source system like Kafka, making it easier for customers to build real-time modern applications and providing a personalized experience. With fully managed Kafka in Confluent Cloud and the scalability and security of Azure, businesses can propel forward in the digital landscape, leveraging data streaming as a critical foundation of modern business.

SEI Investments has been transitioning from a monolithic database architecture to a containerized, event-driven services platform using technologies like Confluent Platform and Apache Kafka since 2017. This shift began with replacing database triggers with Kafka for change data capture (CDC) and has evolved to streaming dozens of applications, allowing the company to meet various business needs such as audit alerts, real-time client data delivery, and system integration. Key to this evolution is the use of Kafka for capturing database events, transforming them into business events, and leveraging the Confluent Schema Registry for data governance. This transformation facilitates breaking down the monolith into modular microservices, enhancing system integration, and offering new business opportunities. Confluent Schema Registry ensures schema evolution is managed effectively, fostering seamless application development and integration. As SEI continues to develop their event streaming capabilities, they aim to further capitalize on their modular architecture to innovate and expand their service offerings.

The pandemic has accelerated the shift to a "remote-first" business approach, prompting companies to enhance collaboration, productivity, and remote work capabilities through modern software design. Confluent and PubNub have collaborated to launch the PubNub Bridge for Apache Kafka with the Confluent Platform, which facilitates real-time data interaction for remote operations without the need for complex IT projects. This tool aims to enhance business value by making event-based data actionable across various industries, including financial services, healthcare, and retail, by providing live alerts, notifications, and real-time actions. The bridge simplifies integrating Kafka data with mobile and web applications, reducing the need for enterprise IT involvement and allowing applications to securely access and broadcast Kafka messages globally. This innovation supports the use of personally owned devices in the workplace and offers personalized and localized customer engagement strategies, ultimately transforming how businesses interact with data and adapt to remote work demands.

Apache Kafka is at the core of a large ecosystem that includes powerful components such as Kafka Connect and Kafka Streams. This article discusses several tools and utilities that can help Kafka developers become more productive, including kafkacat, Confluent REST Proxy, jq, and the Kafka Streams Topology Visualizer. These tools provide various functionalities such as quickly producing and consuming data to and from a topic, easily producing Avro data to a Kafka topic, formatting, manipulating, and extracting data from JSON output of other programs, and creating a graphic representation showing input topics, processing nodes, interim topics, state stores, etc. The article also mentions the release of Apache Kafka 3.8.0, which contains many new features and improvements.

In a discussion on the future of Apache Kafka and software architecture, experts Gwen Shapira, Michael Noll, and Ben Stopford share predictions about the evolution of Kafka and streaming technologies. They foresee significant enhancements in Kafka's capabilities, including the possibility of running clusters with 10 million partitions and the ability to rapidly scale clusters due to architectural changes like KIP-405 (Tiered Storage) and KIP-500 (ZooKeeper removal). These changes are expected to improve scalability, cost efficiency, and performance, enabling quicker autoscaling and better data management. Additionally, the panel anticipates the widespread adoption of streaming features across various products, suggesting a shift towards an "events first" mindset in software architecture. This paradigm shift could lead to multi-paradigm products that incorporate event streaming into traditional database systems, despite challenges in retrofitting older technologies. The conversation underscores the growing importance of event streaming as a dominant paradigm in contemporary software architecture, driven by evolving user expectations and technological advancements in the Kafka ecosystem.

Confluent employs property-based testing to ensure the reliability and efficiency of Confluent Cloud's Tiered Storage feature, which optimizes storage costs and elasticity by transferring data from costly local disks to scalable object storage. This testing strategy complements traditional methods like example-based and integration testing, focusing on critical behaviors such as preventing premature data deletion and ensuring correct data exposure and tiering. The process involves using ScalaCheck’s stateful testing to validate the log-oriented storage system's retention logic, with initial and action generators creating varied test scenarios. Actions like appending records, advancing high watermarks, and simulating segment uploads are tested to ensure the system's integrity and performance. Findings from these tests have led to improved understanding and identification of issues like non-idempotent retention and high watermark mishandling, prompting further enhancements. Future efforts will focus on implementing additional features like retention.bytes and automatic segment rolls to expand the scope of property testing.

Confluent Platform 6.0 introduces significant enhancements to the Confluent REST Proxy, which previously focused on produce/consume use cases, by now allowing Apache Kafka administration via RESTful HTTP APIs. This update enables management of resources such as ACLs, brokers, clusters, topics, and consumer groups, eliminating the need for separate REST Proxy nodes as the APIs are integrated directly into the Confluent Server. This integration simplifies setup and enhances operational efficiency by allowing direct HTTP interaction. Additionally, the update features new APIs that support management tasks like creating topics, updating configurations, and controlling access through ACLs. Looking forward, Confluent aims to improve the REST Proxy with a stateless Consumer API, decoupled serializers/deserializers for producer and consumer APIs, and support for Kafka headers, furthering its utility in diverse programming environments.

Zendesk utilizes Apache Kafka as a foundational service for event distribution across its global pods, which are isolated cloud environments. Each pod has its own Kafka cluster, running on Kubernetes and virtual machines, with communications initially in plaintext. To enhance security and facilitate a global event bus, Zendesk implemented a self-hosted mutual TLS (mTLS) authentication system, using HashiCorp's Vault for certificate management and Consul for key-value storage. The solution includes automated certificate generation, rotation, and monitoring to ensure secure and authenticated Kafka communications. Despite the inability to perform individual certificate revocation due to JVM limitations, Zendesk developed a robust CA root certificate rotation mechanism to maintain system security. The implementation of mTLS comes with a performance impact due to increased CPU usage, primarily from encryption and decryption processes, and the loss of zero-copy optimization. However, the use of kernel TLS is explored as a potential mitigation strategy. Zendesk has also focused on simplifying client onboarding with tools and guides to facilitate the adoption of mTLS across their systems, while ensuring developers can focus on customer value instead of security and compliance intricacies.