September 2015 Summaries

The text discusses the challenges of managing schemas in big data projects, particularly in Hadoop and NoSQL systems. The author shares their experience working on Data Warehouse Offloading projects and how they had to implement a schema management system to enable different teams to use the same data for their applications and analysis. The author highlights the importance of having a shared repository of all schemas used by an organization to make siloed knowledge shared and explicit. They also discuss the benefits of using a schema registry, including enabling stream processing pipelines, improving developer efficiency, and providing a way to load results into HBase or Impala. The text concludes that Confluent's stream data platform includes a Schema Registry as a first-class application, which is crucial for building production-ready data pipelines.

Apache Kafka is a high-throughput distributed messaging system used across various industries by thousands of companies, including Twitter, Netflix, and Cisco. It has become a mission-critical infrastructure component for modern data platforms. The upcoming Apache: Big Data Europe conference will feature 120+ sessions, including talks about Apache Kafka and stream processing. Confluent's Michael G. Noll will present "Being Ready for Apache Kafka: Today’s Ecosystem and Future Roadmap," providing an update on the growth of the Kafka project community and what's required to put Kafka in production. The presentation will cover the current ecosystem, operational tools, peripheral components, and the upcoming project roadmap. Confluent is also hosting Current 2024 in Austin, Texas, where attendees can discuss Apache Kafka and Apache Flink with tech leaders and industry giants.

Apache Kafka has become the largest deployment of Apache Kafka in production at any company, with LinkedIn's deployment surpassing 1.1 trillion messages per day, and is used by hundreds of other companies including web giants like Netflix, Uber, and Pinterest, as well as large enterprises like Cerner, Cisco, and Goldman Sachs.

The upcoming fall conference season offers numerous opportunities for developers to learn about Apache Kafka, stream processing, and other related topics. Conferences such as Strata + Hadoop World New York and ApacheCon: Big Data Europe provide a platform for networking with the open source communities and learning from industry experts like Jay Kreps. The release of Apache Kafka 3.8.0 includes several new features and improvements, including a JavaScript client that can be used to build applications that interact with Kafka topics. Additionally, conferences offer a chance to meet the Confluent team and learn about best practices for using Kafka in production, as well as how to avoid common mistakes.