November 2019 Summaries

The text discusses the potential advantages of hybrid cloud architectures, emphasizing the importance of securely moving data across different environments. It highlights the role of event-driven architectures, particularly those using Apache Kafka, in addressing the complexities and costs associated with hybrid cloud deployments. By adopting an event-first mindset, organizations can benefit significantly in terms of operational complexity and cost management. The text illustrates how Kafka's durability and scalability make it an effective bridge between cloud and on-premise environments. It also explores the use of Confluent Replicator to create a replicated event nervous system, reducing security and complexity risks. The text provides a hypothetical example of an electronic stock trading system transitioning to a hybrid cloud setup, optimizing costs and maintaining service-level agreements. Additionally, it mentions a demo using Kubernetes and Confluent Operator to deploy Confluent Platform in a Google Kubernetes Engine cluster, showcasing the ease of creating a multi-cloud environment and the benefits of using Confluent Cloud's managed services. The demo is intended as a learning tool, not a production solution, and encourages experimentation with replicated event streams in hybrid cloud deployments.

The text discusses Apache Kafka, a distributed streaming platform, and its integration with ksqlDB, an event streaming database. The authors highlight the benefits of using Kafka for data processing, buffering, and backpressure, as well as its decoupling capabilities. They also discuss stream-table duality, which allows converting relational data into events, enabling real-time processing and enrichment. The text introduces Kafka Streams, a native component of Apache Kafka, and ksqlDB, an event streaming database built on top of Kafka. Both technologies offer ease of use, security, and deployment benefits, with ksqlDB providing a SQL-flavored abstraction layer for stream processing applications. The authors emphasize the importance of understanding the stream-table duality concept and choosing between ksqlDB and Kafka Streams based on resources, team aptitude, and use case. They also highlight the potential of ksqlDB to become a powerful part of the Confluent Platform and its future developments.

The new release of ksqlDB, formerly known as KSQL, introduces significant enhancements to stream processing by integrating traditional database-like functionalities, such as pull queries and connector management, with the capabilities of Apache Kafka® streams. These pull queries allow users to perform point-in-time lookups on materialized tables, complementing the existing continuous "push queries" that provide real-time data streams. The integration simplifies the architecture needed for event streaming applications, reducing reliance on multiple systems by consolidating processing and querying capabilities within ksqlDB. This evolution aims to make event streaming as straightforward as building traditional CRUD applications by offering an interactive and unified experience for developing and managing event-driven solutions. While ksqlDB does not aim to replace primary storage systems like Postgres or analytical stores like Elasticsearch, it excels in providing on-the-fly materialized views and seamless integration with other data stores through its enhanced connector management.

Unity's data team, despite being small, is responsible for managing the data infrastructure that supports both its development platform and monetization network, which includes handling half a million events per second and millions of dollars in transactions without outages since implementing Confluent Platform and Apache Kafka®. Initially, Unity faced challenges integrating various departmental data pipelines across different technology stacks, but successfully unified them with Kafka, particularly during a significant cloud migration from AWS to GCP. Supported by Confluent, Unity transitioned from a batch processing to an event streaming model, drastically reducing data latency from two days to 15 minutes, which has facilitated numerous business improvements and enabled real-time decision-making. This transformation has led to widespread adoption of event streaming systems across Unity and interest in using Kafka for real-time machine learning model training, emphasizing the platform's reliability and the shift in internal perceptions towards more stable and efficient data handling practices.

Kafka Connect is a configuration-driven tool for reliable, scalable, distributed streaming integration between Apache Kafka and other systems. It has connectors for many systems and supports custom connectors through its API. The Kafka Connect API provides Single Message Transforms (SMTs) that operate on every single message in the data pipeline as it passes through the connector. These transforms can be used to manipulate records before writing them to a Kafka topic or after reading from a sink system. They are classified into sources and sinks, which pull or push data respectively. The transforms can be chained together for more complex operations. However, they should only be used for simple, limited mutations of the data and not for heavy processing or external API calls. The transformations are compiled as JARs and configured in the connector properties. They support both record keys and values, allowing manipulation of all parts of the ConnectRecord.

Confluent Cloud is now available on Microsoft Azure, offering true freedom of choice for public cloud developers who want managed Apache Kafka as a service. With Confluent Cloud, developers can focus on building applications rather than managing infrastructure, and enjoy cloud-native capabilities to quickly harness the power of Kafka. The platform provides a complete event streaming and data modernization solution, including fully managed Kafka, Schema Registry, and ksqlDB, and allows users to scale elastically without sizing and provisioning clusters. Confluent Cloud offers consumption-based pricing, completely configurable dedicated setups, and private networking options, making it an attractive option for organizations looking to improve customer experience, scale their businesses, and sharpen their competitive edge.