January 2018 Summaries

In designing a topic structure for Apache Kafka, it's essential to balance performance considerations with data integrity and modeling aspects. The common wisdom is to group events of the same type in the same topic, but this approach can lead to issues when using different topics for various event types. Instead, consider grouping events based on their logical relationships or business requirements. For example, if you're handling customer-related events, it's crucial to maintain order and ensure that related events are processed sequentially. Using a timestamp-based ordering approach is not sufficient in stream processes, as clocks can be desynchronized. Confluent Schema Registry provides flexibility in handling multiple event types in a single topic, especially with the new configuration options for constructing subject names. Ultimately, the goal is to create a topic structure that balances performance and data integrity while supporting your specific use case.

Apache Kafka and Apache Mesos are two well-known projects that have undergone significant changes since the last blog post from Confluent in July 2015. The Microservices paradigm offers benefits like autonomous teams and agile development, but also creates challenges such as increased communication between distributed instances and the need for orchestration. Confluent's resources can help build a scalable microservice architecture using Kafka, its Streams API, and Mesos. Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks, while DC/OS is a Mesosphere-backed framework on top of Apache Mesos. A practical example shows how to set up a Kafka and microservices infrastructure on DC/OS, leveraging the benefits of both systems. The Kafka Streams API allows for lightweight but scalable stream processing logic, which can be added to any JVM application or microservice. Mesosphere's DC/OS enables management of both Kafka infrastructure components and Kafka Streams microservices, providing automated provisioning, upgrading, elastic scaling, unified management, and monitoring.

The Kafka Summit in London will be a premier streaming systems event for data architects, engineers, DevOps professionals, and developers, featuring a deeply technical conference with community-focused discussions. The summit is expected to bring together Apache Kafka practitioners to share experiences and best practices, while also showcasing related events such as tutorials and advanced training sessions. The full agenda can be found on the event website, and early-bird pricing ends January 26th. Register now to save £100 off conference fees. Additionally, Confluent and Amazon Web Services have collaborated to make cloud services and data management easier for government agencies.