June 2018 Summaries

The blog post explores the integration of Apache Kafka, Kafka Connect, and KSQL to manage and analyze real-time data from Ubiquiti network devices, using MongoDB as the primary data source. It demonstrates how Debezium, a change-data-capture tool, streams data from MongoDB into Kafka, creating a live mirror for reference data without needing convoluted lookups. The process involves setting up Debezium, exploring data with tools like Robo 3T, and using KSQL to create streams and tables that enrich Ubiquiti syslog events with metadata from access points and user devices. By rekeying Kafka topics and employing SQL queries, the enriched data is visualized in Elasticsearch with Kibana and used for event-driven alerts. The post emphasizes the ease of building scalable real-time stream processing applications with KSQL, leveraging Kafka Connect for seamless data integration and offering insights into using these tools for powerful analytics.

Confluent Cloud is a fully-managed streaming data service based on open-source Apache Kafka that allows developers to build mission-critical streaming applications with ease, without the operational burden of managing their own clusters. The service provides a single source of truth and supports hybrid deployments across multiple cloud providers and on-premises environments. To design and manage these applications, Confluent offers tools such as KSQL for stream processing, Confluent Replicator for data migration, and Confluent Control Center for monitoring. Developers can also leverage the service to enhance stateful processing capabilities with versioned key-value state stores. By using Confluent Cloud, developers can accelerate building enterprise-ready streaming platforms in the cloud.

Confluent has launched Confluent Hub, a centralized platform designed to enhance the Apache Kafka and Confluent Platform community by providing easy access to components necessary for building efficient streaming data pipelines and event-driven applications. Confluent Hub simplifies the process of finding, reviewing, and downloading Kafka Connect connectors, transformations, and converters, alleviating the need for manual builds and installations. It also introduces a Confluent Hub client tool that streamlines component installation into Confluent Platform deployments. The platform supports various integrations and is open for community contributions, encouraging developers to share their components within the ecosystem. Confluent Hub aims to improve productivity in data streaming by offering a more accessible and functional discovery mechanism for users. Additionally, Confluent has been recognized as Microsoft's 2024 OSS on Azure Global Partner of the Year, highlighting its dedication to delivering open-source solutions on Azure.

The Confluent Platform May 2018 Preview release introduces powerful new capabilities for KSQL and the Schema Registry UI, designed to help operations teams manage schema and allow beginners to learn about it. A new Schema Registry UI allows users to view schema per topic, version history, and compare between previous and current schemas. An autocomplete feature has been added to the KSQL query editor to compose queries faster, while a new INSERT INTO statement lets users write query output into an existing stream. Docker images are now available for preview versions of KSQL server and CLI, allowing for easier deployment. The DROP statement supports deleting underlying Kafka topics and registered Avro schemas, ensuring clean-up during iterative development and testing.