March 2020 Summaries

Bolt, a prominent ride-hailing app, uses ksqlDB in its data warehouse stream processing pipeline to synchronize and replicate data from various sources for analysis and decision-making. The system allows data from transactional databases, mobile applications, and other external integrations to be centralized, facilitating easy access for analysts and decision-makers to query, compare, and predict across different business lines and time periods. ksqlDB, chosen for its seamless integration with the Kafka ecosystem, enables efficient data replication by creating streams that transform raw data from source topics to destination topics for ingestion into the data warehouse. Built on Kafka consumer and producer protocols, ksqlDB supports horizontal scaling and offers functionalities like data filtering, stream merging, and distribution, eliminating the need for new Java-based Kafka Streams applications. This approach has allowed Bolt to replicate over 500 topics to its warehouse, with plans to explore stateful stream processing in the future.

The document provides a detailed account of the process and considerations involved in creating a Kafka connector to integrate OpenSky Network's air traffic data into Apache Kafka, and subsequently using it for data visualization and analytics. The author shares insights and challenges encountered, emphasizing the benefits of using the Kafka Connect framework for scalability, fault tolerance, and ease of configuration compared to writing a custom producer. The text outlines practical steps for developing a connector, including schema modeling, licensing considerations, and testing strategies, while also highlighting the use of additional tools like ksqlDB for querying and Elasticsearch and Kibana for visualizing geospatial data. It stresses the importance of proper configuration and testing to ensure efficient data handling and encourages sharing successful connectors with the Apache Kafka community through platforms like Confluent Hub. The document also touches on advanced topics such as user-defined functions in ksqlDB and the integration of Elasticsearch for enriched data analytics, offering a comprehensive guide for developers looking to leverage Kafka Connect for data integration projects.

The text explores the evolution of data analytics architectures, comparing state orientation with event orientation, and highlights the integration of both paradigms in modern systems. It delves into the historical context of human evolution to draw parallels with technological advancements, emphasizing the transition from linear to branching models in both fields. The piece discusses the technical implementations of state and event orientations, noting the complexities introduced when both are combined in modern Internet of Things (IoT) and big data platforms. The article introduces ksqlDB, a database for event streaming developed by Confluent, which combines real-time querying with event streaming capabilities, offering advanced features like high performance and fault tolerance. It illustrates how ksqlDB can be used to manage sensor data in IoT applications, allowing both traditional pull queries and real-time event subscriptions. The text concludes by positioning ksqlDB as a significant step in the technological evolution of data systems, questioning whether it serves as a bridge between big data and fast data systems.

Confluent Cloud's architecture is composed of a control plane and data plane. The data plane includes a network architecture that abstracts away individual brokers, allowing for dynamic scaling without requiring configuration changes in clients. A load balancer is used to proxy traffic to a randomly chosen, healthy broker, and an additional layer 4 load balancer provides abstraction from the brokers altogether. The control plane consists of a mothership and satellites, which communicate via Kafka. The mothership receives API requests and makes scheduling decisions, while satellites manage the lifecycle of physical clusters and report their status to the control plane. A physical stateful cluster (PSC) declaratively describes the desired state of a physical cluster, and the Confluent Operator materializes this into Kubernetes-native objects. Confluent Cloud has benefited Kafka by providing a cloud-first organization, allowing for rapid testing and deployment of new features, and enabling improvements to Kafka itself, such as improved fault tolerance and scalability. The architecture has also helped to learn valuable lessons about API design, third-party software dependencies, and cost modeling, which can be applied to other projects.

The latest release of ksqlDB, a popular open-source data streaming platform, introduces several new features that improve its capabilities. The retention time for windowed aggregates can now be configured in the aggregation statement itself, allowing for more flexibility and control over storage space. Additionally, the grace period for late-arriving events in windowed computations can also be configured, enabling applications to handle delayed events more effectively. Furthermore, ksqlDB 0.8.0 introduces a new built-in function called LATEST_BY_OFFSET, which allows users to track the latest value of a column when aggregating events from a stream into a table. The release also includes improvements to the client and server API, making it easier for developers to work with ksqlDB. Other notable features include two new built-in functions, REGEXP_EXTRACT and ARRAY_LENGTH, as well as various bug fixes and other enhancements.

Enterprises are increasingly moving their applications and ETL pipelines to the cloud, resulting in a need for real-time cloud ETL solutions that can span multiple cloud providers. This blog post outlines a use case involving a log ingestion pipeline leveraging Confluent Cloud, which manages Apache Kafka brokers, Kafka connectors, Confluent Schema Registry, and ksqlDB. The pipeline demonstrates how developers can automate the ingestion process using the Confluent Cloud CLI, enabling seamless data flow between different cloud environments such as AWS, Azure, and Google Cloud. By configuring connectors for data sources like Amazon Kinesis and sinks like Google Cloud Storage, enterprises can efficiently handle multi-cloud requirements driven by business regulations, cost considerations, and cross-company acquisitions. This setup allows developers to choose applications based on their specific needs, independent of organizational cloud preferences, while Confluent Cloud serves as a central hub for managing data across cloud platforms. The post also suggests utilizing a promotional code for additional Confluent Cloud usage and highlights the benefits of using Confluent's managed services for enhanced security and integration capabilities.

Confluent offers two approaches to multi-datacenter replication: Multi-Region Clusters and Confluent Replicator, which can be used for both single-cluster and multi-cluster deployments. Multi-Region Clusters are a single cluster spanning multiple datacenters, while Confluent Replicator operates on two separate clusters. Confluent Replicator provides a high-performance and resilient way to copy topic data between clusters. It enables bidirectional replication, fan-out, and reduces the number of public communication channels between hybrid cloud environments. Confluent Control Center integrates with Kafka Connect, enabling the creation and management of connectors, including Replicator, using a graphical interface. The control center provides detailed monitoring for Replicator, ensuring efficient replication flows. Replicator includes features such as offset translation, cyclic replication prevention through provenance headers, and topic synchronization to ensure consistent configuration state between source and destination clusters. It can be deployed into existing Kafka Connect clusters or run as a standalone process with fault tolerance and load distribution capabilities.

Confluent has enacted its business continuity plan in response to the COVID-19 pandemic, ensuring the health and well-being of customers, employees, and the community. The company remains committed to delivering exceptional customer experiences while maintaining operational stability. Confluent has also achieved FedRAMP Ready status for its Confluent Cloud for Government offering, demonstrating a commitment to rigorous security standards. Furthermore, an expanded partnership between Confluent and Databricks aims to simplify data integration and governance, enabling enterprises to create value more efficiently.

Event stream processing is a powerful technology that solves many business challenges, including big data ingestion, data integration, and real-time data processing. It allows for the analysis of big data streams in-motion as it's generated, opening up new use cases for organizations across various industries. Two popular tools for building event streaming applications within the Apache Kafka ecosystem are ksqlDB and Kafka Streams. While both provide solutions for stream processing workloads, they have key differences to be aware of. ksqlDB is an event streaming database built on top of Kafka Streams that provides a high-level SQL syntax for building streams and tables, creating materialized views, and managing Kafka connectors without worrying about traditional application management concerns. Kafka Streams, on the other hand, is a Java virtual machine client library for building elastic and fault-tolerant stream processing applications on top of Kafka. It allows developers to build stream processing workloads with the full power of any JVM-based language. The choice between ksqlDB and Kafka Streams often boils down to trade-offs between control and simplicity. By using these technologies, organizations can harness the power of real-time data processing to power their AI projects with real-time data.

The decision to cancel the Kafka Summit London due to COVID-19 concerns reflects a commitment to public health, despite the disappointment it causes to presenters, attendees, and the Apache Kafka community. Efforts are being made to refund fees to registered participants and to accommodate speakers by encouraging them to submit their presentations for the upcoming Kafka Summit Austin. Locally, Meetups are being assessed for safety, with some being converted to online formats in accordance with health guidelines. The organization looks forward to future gatherings, including the Kafka Summit Austin in August, while encouraging engagement with online content like AI Day's Keynote and upcoming events such as AWS re:Invent 2024, which will explore data streaming and integration innovations.

Confluent is a company that provides an event streaming platform, enabling enterprises to build new applications leveraging real-time data. The company was founded by the original creators of Apache Kafka and has gained widespread adoption in various industries. Confluent's culture values product-led growth, empowering employees to make decisions and drive impact while growing their careers. The company celebrates International Women's Day and recognizes its female leaders who share their experiences joining the company, making an impact, and shaping its culture. With a strong promote-from-within culture, flexible work arrangements, and a focus on employee well-being, Confluent offers a unique opportunity for individuals to grow and make a difference in the industry.

The Elasticsearch sink connector helps integrate Apache Kafka with Elasticsearch, allowing users to stream data stored in Kafka into Elasticsearch for log analysis or full-text search. This integration can also be used for real-time analytics and other applications like Kibana. The Elasticsearch connector pushes events from Apache Kafka into an Elasticsearch index. It supports Elasticsearch 2.x, 5.x, 6.x, and 7.x versions. The tutorial guides users through the process of sending JSON documents from Kafka into Elasticsearch using ksqlDB for some Kafka operations. The connector configuration includes attributes like "connector.class", "tasks.max", "topics", "name", "connection.url", and "type.name". Users can also update the configuration to include a JSON converter or ignore keys and schemas, depending on their requirements. The tutorial further explains scenarios where product listings data is inserted into a MySQL database that needs to be sent into Elasticsearch for search functions. It demonstrates how to create a JDBC source connector and an Elasticsearch sink connector in such cases. Additionally, it discusses the use of TimestampRouter SMT to rotate indexes daily and handle tombstone messages. Lastly, the tutorial emphasizes the importance of tuning parameters like flush.timeout.ms, max.buffered.events, and batch.size for optimal performance from the Elasticsearch connector. It also mentions that Confluent provides a variety of sink and source connectors for popular databases and filesystems to stream data in and out of Kafka.

The document discusses the challenges of minimizing the "blocked factor" in API-driven web application development, highlighting the trade-offs between flexibility and usability in integrating external APIs. It introduces Mox, a tool developed by Confluent to address these challenges by combining the functions of a proxy server and a mocking framework. Mox allows developers to manage API requests flexibly, providing the ability to mock or modify requests as needed, thereby helping to reduce time spent blocked by unavailable backend services. The text details how Mox can be used to simulate various API behaviors, aiding in both feature development and bug investigation. It also outlines the evolution of Confluent's approach from relying on mock servers to a hybrid strategy that balances using live APIs with mock APIs, thus optimizing workflow efficiency and reducing maintenance overhead. Additionally, the document briefly mentions the release of Confluent Platform 7.8 and the content shared at Current 2024, emphasizing the importance of data streaming in modern engineering.