August 2016 Summaries

The text discusses the transformative potential of distributed systems, emphasizing their ability to extend beyond mere scalability to operate across entire companies or data centers, thus addressing new challenges and enabling multi-tenancy. It highlights the evolution from single-application databases to distributed systems that can integrate and manage data across many applications, citing Apache Mesos and Apache Kafka as examples of systems that have successfully expanded their scope. The piece underscores the importance of both scalability and multi-tenancy, noting that while scalability is often celebrated, achieving effective multi-tenancy is more challenging and crucial for supporting numerous users and use cases within an organization. It describes the enhancements made to Kafka, such as security features and quotas, which have allowed it to be deployed as a company-wide data pipeline in large organizations like Netflix and Uber, and concludes with a nod to an educational series on building scalable applications using these technologies.

This blog post is part of a series examining the Streams API of Apache Kafka, focusing on a new application reset tool designed to simplify the process of reprocessing input data from scratch in Kafka Streams applications. The tool, introduced with Confluent Platform 3.0.1 and Apache Kafka 0.10.0.1, addresses the complexities and potential errors of manually resetting applications by managing global resets, while local resets are handled within the application code using KafkaStreams#cleanUp(). The post details how to prepare an application for reset, the steps involved in using the reset tool, and the importance of managing offsets and internal state to ensure accurate reprocessing. It also highlights the drawbacks of using workarounds like changing the application ID, emphasizing the need for proper cleanup of internal topics, state directories, and metadata to avoid resource wastage. The article concludes with an encouragement to try the new tool and provides resources for further learning about Kafka Streams.

This past April, Confluent hosted the inaugural Kafka Summit in San Francisco, bringing together the entire Kafka community to share use cases, learnings, and participate in a hackathon. The summit provided valuable content for Kafka users to share their experiences with the technology. Jay Kreps, original co-creator of Kafka, delivered a keynote talk on the shift from batch-oriented data engineering to a digital business based on stream data. Other popular sessions covered topics such as large-scale stream processing with Kafka Streams, real-time systems and event-oriented data platforms using Kafka, and best practices for configuring Kafka. The summit also featured success stories like Uber's use of Kafka in its real-time transit infrastructure. Confluent plans to release more information about the 2017 event soon, which promises new opportunities for community members to meet and exchange ideas.