April 2017 Summaries

Apache Kafka is a popular open-source event-driven stream processing platform used by companies like LinkedIn, Goldman Sachs, Target, Pivotal, Heroku, Confluent, and others. The Systems track at Kafka Summit NYC will focus on learning how to get the most out of Apache Kafka, with community and industry experts sharing the latest features and best practices for running and scaling Kafka in production. Speakers from these companies will share their experiences, lessons learned, and new features used, addressing challenges faced when running Kafka at scale and providing solutions along the way. The track is designed to inspire and guide attendees as they work with Kafka in their own organizations, covering topics such as security, scalability, and automation. With experts like Anton Gorshkov from Goldman Sachs and Apurva Mehta from Confluent presenting on topics like Exactly Once Semantics and security features, this track is a must-attend for anyone running Kafka at scale. The event will also feature other notable sessions and provide an opportunity to network with the community of Kafka experts across various industries.

In this post, Confluent shares three common pitfalls developers face when operating Apache Kafka in their organizations. These include under-replication issues, blind restarts causing data loss and network saturation, and improper partition reassignment leading to data movement challenges. The company emphasizes the importance of understanding root causes for these issues and using tools like Confluent's auto data balancer to mitigate risks. By sharing lessons learned from real-world experiences, Confluent aims to help developers avoid common pitfalls and keep their Kafka systems running smoothly.

Apache Kafka is being used by Pandora, a data-driven company, to power its real-time stream processing analytics and ad trafficking infrastructure. The company has migrated from a legacy batch processing event ingestion pipeline to a system designed around Kafka, which provides an opportunity for streamlined event processing and security controls. To address schema evolution challenges, Pandora integrated the Kafka Connect API and Confluent Schema Registry into its data pipeline, enabling efficient real-time copying of data between Kafka and other systems like HDFS. The company has also implemented a redundant Disaster Recovery (DR) pipeline to ensure business continuity in case of failures, and has tuned connector configurations for optimal performance. With the benefits provided by Kafka Connect API and Confluent Schema Registry, Pandora has built a robust data pipeline with production readiness from Ad Servers to HDFS using Kafka.

Kafka Summit NYC is an upcoming event where industry professionals will gather to discuss the use of Apache Kafka in building integrated data architectures and expanding cloud use cases, with the Streaming Pipelines track focusing on sharing practitioner experiences and compelling Kafka use cases. The program committee, responsible for selecting track sessions, aimed to choose talks that would appeal to a significant portion of attendees and cover diverse aspects of the Kafka ecosystem, ultimately narrowing down over 100 submissions to just 8 per track. Highlighted sessions include Airbnb's use of Kafka for reliable logging, Yelp's real-time data pipeline, and Ancestry’s approach to data integration. This event, featuring the largest gathering of Kafka experts, will be followed by Current 2024 in Austin, Texas, focusing on discussions around Apache Kafka and Apache Flink.