May 2019 Summaries

Part 3 of the Spring for Apache Kafka Deep Dive blog series introduces Spring Cloud Data Flow, a toolkit designed to enhance developer productivity by simplifying the development, deployment, and orchestration of event streaming pipelines using Apache Kafka. The blog explains how Spring Cloud Data Flow manages data pipelines from design to production deployment, supporting both real-time event streaming and short-lived task/batch applications. Key components include Spring Cloud Skipper for handling application lifecycle operations and Micrometer-based monitoring with customizable Grafana dashboards. The blog provides guidelines for setting up a local development environment with Docker, illustrating the creation and deployment of event streaming pipelines using Stream DSL syntax. Additionally, it highlights the integration of Kafka Streams applications within these pipelines and offers insights into monitoring, security, and operational auditing. Concluding with a preview of future topics, the blog sets the stage for exploring advanced event streaming topologies and continuous deployment patterns in subsequent parts.

In this webinar series, Red Pill Analytics presents an overview of their work with BMW Group on developing an omnichannel transformation using data streaming. The first part discussed the implementation of an event streaming architecture for a customer using Apache Kafka®, KSQL from Confluent, and Kafka Streams. In this second part, they discuss the challenges faced during development, building, and deployment of the KSQL portion of their application, and how Gradle was used to address them. They also explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. The series concludes with a discussion on writing a KSQL event streaming application, including what scripts look like, how they are organized in a Git repository, and how they combine them into pipelines.

To avoid operational mistakes when self-managing the Schema Registry component in an architecture, it is recommended to configure it properly from the start and manage it well. This includes deploying Schema Registry instances on their own hosts, using a single global cluster across the entire company or geographical areas, and configuring different settings between instances that should be the same. Additionally, proper security measures such as SSL encryption, SSL or SASL authentication, and monitoring are crucial to ensure the service is healthy and able to service clients. Mistakes like co-locating Schema Registry with other services, creating separate instances within a company, and not backing up the schemas topic can lead to duplicate schema IDs, lost schemas, and inaccessible services. Furthermore, improper configuration settings, such as mixing election modes or mis-configuring SSL keys, certificates, keystores, or truststores, can also cause problems. By following best practices and avoiding these operational pitfalls, developers can build GenAI apps faster and more efficiently.

Kafka Summit London recently took place with a significant increase in attendees compared to last year. The event featured keynotes, emceeing, and various sessions across four tracks - Core Kafka, Event-Driven Development, Stream Processing, and Use Cases. Videos of the session are now available online for those who missed it or want to revisit them. Confluent Community Catalyst Program is accepting nominations for individuals who have made outstanding contributions to the Confluent or Kafka communities. The next event will be in San Francisco on September 30-October 1, with an agenda set to launch soon. In other news, data streaming world will gather in Austin, Texas for Current 2024 from September 17-18, focusing on Apache Kafka® and Apache Flink®. Confluent and Amazon Web Services (AWS) have partnered to simplify the transition to and management of cloud services.

In this emerging machine-to-machine (M2M) architecture, MQTT, Apache Kafka, and Scylla work together to provide an end-to-end IoT solution. IoT is a fast-growing market, expected to reach $6.5 trillion by 2024, with over 25 billion connected devices in 2018. The goal of this demo is to demonstrate an end-to-end use case where sensors emit temperature and brightness readings to Kafka and the messages are then processed and stored in Scylla. MQTT is used for device communication, while Apache Kafka handles high-velocity data ingestion, and Scylla provides a scalable, distributed, peer-to-peer NoSQL database that works as a drop-in replacement for Cassandra. The demo uses Confluent MQTT Proxy, Kafka Connect Cassandra connector, and Mosquitto to simulate MQTT sensor activity and publish messages to the corresponding topics in Kafka. With the alignment between MQTT, Kafka, and Scylla, application developers can easily build solutions that harness IoT-scale fleets of devices and store the data from them in Scylla tables for real-time as well as analytic use cases.

To build GenAI apps faster, consider using event streaming microservices patterns, where events with schemas are published by services and subscribed to by other services, allowing for loose coupling and easy extension. Schemas serve as contracts between services, defining the structure of data exchanged between them, ensuring compatibility and preventing breaking changes. A schema registry, such as Confluent Schema Registry, is used to manage and validate schemas, providing a framework for developers to specify, document, evolve, and validate schemas, and enabling automated schema validation and testing. This approach enables fast development, reduces coupling between services and teams, and ensures data quality by preventing incompatible events from being produced or consumed in production.

Red Pill Analytics recently collaborated with a Fortune 500 e-commerce company to transform their inventory management system using data streaming. The company had traditionally used large warehouses for shipping, but these were slow and couldn't keep up with modern strategies like same-day dropshipping or industrial vending. They planned to roll out smaller, strategically located distribution centers and selected Oracle Warehouse Management Cloud (Oracle WMS Cloud) as their new warehouse management system. Red Pill Analytics was hired to design and implement data integration processes required to connect Oracle WMS Cloud with the company's on-premises systems, including PeopleSoft, an in-house order management system, and a legacy warehouse management system. The customer had already invested heavily in MuleSoft®, which helped abstract many of these different sources/targets as simple REST APIs. Red Pill Analytics chose to use Apache Kafka® and the Confluent Platform for data streaming due to their ability to operate just-in-time and efficiently manage inventory levels, delivery times, and approaches. They used Oracle GoldenGate for Big Data 12c (OGG for BD) to deliver relational data change events to Kafka and MuleSoft to consume relevant topics and translate them into required API calls. The final solution involved using KSQL primarily as the functional engine for expressing streaming data transformations, and Kafka Streams for the final packaging of payloads before shipping them off to specific APIs.

The Apache Kafka and Confluent communities are comprised of hundreds of thousands of people worldwide who actively contribute to the technology and community through writing code, tutorials, presentations, and answering questions. The Confluent Community Catalyst Program is a recognition initiative for individuals who invest their time and energy in the community, making it a habit of contributing knowledge, enthusiasm, support, encouragement, mentoring, and code. To be eligible, candidates must demonstrate expertise in Apache Kafka and/or the Confluent Platform, as well as actively sharing their knowledge or contributing to projects within the community. The program aims to identify individuals who are making a significant impact on the community and will select a new class of Catalysts each year.

Confluent Cloud has made significant strides in providing a cloud-native event streaming service, enabling users to scale their Kafka workloads instantly without provisioning or sizing clusters. This allows for elastic scaling from 0 to 100 MB/s and tens of GB/s with provisioned capacity, while paying only for the data streamed to Kafka. The service also introduces Confluent Schema Registry, Kafka S3 sink connector, and ksqlDB as fully managed services, addressing gaps in managed Apache Kafka services. With transparent consumption-based pricing, users can get a realistic production cluster for $50 per month with no commitment or fees. This marks a key step forward in providing a complete event streaming platform for building contextual event-driven applications that power today's event-driven enterprise.

The text discusses building a payment system using an event-driven architecture, specifically with Confluent, Flink, and MongoDB. It highlights the benefits of event-first thinking, which enables the creation of a generic, data-centric, distributed application runtime that can be used to build various applications. The authors argue that transactions provide guaranteed behavior but are pushed down into database runtime, whereas events offer more control and flexibility. They introduce four pillars for building scalable development: business function, instrumentation plane, control plane, and operations plane. These pillars enable the creation of a system that is simple, scalable, and maintainable. The authors demonstrate how to build a payment system using these pillars, including event streaming, aggregation, and control planes. They also discuss the importance of DevOps and the need for a strong instrumentation, control, and elasticity story.

In the context of event streaming data pipelines built on Apache Kafka, multiple clusters of Kafka Connect and KSQL are often used for various reasons including managing different teams or handling specific use cases such as stream processing applications and analytics. Confluent Control Center is a tool that enables interaction with these clusters using a GUI, allowing users to select from multiple clusters and manage their configuration. The Confluent Platform is a comprehensive solution for building, monitoring, and managing event streaming pipelines on Apache Kafka, providing features such as enhanced security, support for Apache Flink, and integration with Amazon OpenSearch through OpenSearch Ingestion.

The text discusses how Apache Kafka's consuming model works, focusing on its distributed systems challenges like data consistency, failover, and load balancing. It explains the notion of consumers and consumer groups, as well as the role of Kafka brokers in managing consumer groups. Additionally, it delves into the rebalance protocol that ensures fault tolerance against consumer application failures. The text also mentions recent improvements to this protocol and highlights some features of Apache Kafka 3.8.0.

Confluent Platform 5.2 has introduced features that make it easier to manage large Apache Kafka clusters, including improved configuration management and dynamic broker reconfiguration, which allows for changes to be made without restarting brokers, reducing time savings in troubleshooting misconfigured clusters. Control Center can now comfortably handle around 120,000 individual partition replicas, allowing it to manage up to 40,000 individual partitions, making it more suitable for large deployments with typical topic partition counts of six, covering over 90% of existing customer workloads.

Confluent continues to innovate and expand its capabilities, having grown from an idea into an industry pioneer in just four years. The company's vision of event streaming as a backbone of modern applications has gained traction among over 60% of the Fortune 100 companies, with many adopting this paradigm shift. Confluent's KSQL tool has become one of the most popular tools in the Kafka ecosystem, and the company is now announcing its partnership with PipelineDB to further enhance its capabilities. This partnership brings together two teams sharing a vision of a future powered by event streaming, and will help complete the event streaming platform journey started by Confluent years ago. The company's recognition as Microsoft's 2024 OSS on Azure Global Partner of the Year reflects its commitment to delivering outstanding open source-based applications and infrastructure solutions on Microsoft Azure.