December 2017 Summaries

The December release of KSQL 0.3 for Apache Kafka introduces significant updates, including support for Avro data format integration via the Confluent Schema Registry, allowing automatic schema inference to simplify data handling. This version enhances conversion capabilities between Avro, JSON, and delimited formats, streamlining real-time ETL processes and facilitating seamless joins across different data formats. Observability and operability have been improved with new metrics for streams, tables, and queries, including detailed statistics and failure metrics through the DESCRIBE EXTENDED and EXPLAIN statements. Additionally, KSQL servers now start up faster and utilize resources more efficiently, with features to avoid redundant query processing. The release is a call to the community for continued collaboration, with future enhancements planned for 2018 to further establish KSQL as a premier data processing tool for Kafka.

Monitoring helps provide assurances that all your services are working properly, meeting SLAs, and addressing business needs. When monitoring a Kafka deployment in real-time, it's essential to focus on what the business cares about, such as application-level visibility, stream latency, consumer group membership, partition assignment, message delivery statistics, and performance bottlenecks. A common pitfall of generic monitoring tools is that they import all available metrics from various systems into a metrics swamp, which cannot produce valuable insight without Kafka context or expertise to determine important metrics. Instead, a domain-specific monitoring solution designed for Kafka is necessary to provide real-time visibility into the wellbeing of the Kafka deployment and identify critical issues. Confluent Control Center provides real-time monitoring capabilities, including stream latency, consumer group membership, partition assignment, message delivery statistics, and performance bottlenecks, with features such as end-to-end stream monitoring, stream delivery, and request lifecycle analysis. It also inherits performance and scalability improvements from Kafka, provides a 2-for-1 efficiency in operations, offers a unified security configuration experience, and supports enterprise-level support through the Confluent subscription.

The blog post is the final installment in a series exploring exactly-once semantics in Apache Kafka, particularly focusing on how these semantics are implemented in the Kafka Streams API. It explains the complexity of achieving exactly-once processing in real-time stream applications and describes how Kafka's transaction feature, introduced in version 0.11.0, enables this guarantee by allowing message processing to be atomic. This is achieved through the Kafka Streams library, which uses transactions to ensure that data updates, offset commits, and message outputs are synchronized, thus preventing issues like duplicate writes and processing. The post emphasizes the ease of enabling exactly-once semantics by simply configuring a setting, without requiring code changes, and highlights the importance of this feature in maintaining application correctness under failure scenarios.

Confluent Control Center is a specialized monitoring solution for Kafka deployments, designed by the creators of Kafka to ensure that mission-critical applications run smoothly and meet service level agreements. The tool offers key features such as system health monitoring, end-to-end stream monitoring, and alerting, allowing users to verify the performance of their Kafka clusters and applications effectively. This blog post introduces the general availability of Confluent Platform 7.7, highlighting new features including enhanced security with OAuth support, integration with Apache Flink, and a new connector. It also provides resources such as video tutorials and a demo GitHub repository for users to explore Confluent Control Center's capabilities in monitoring Kafka, and discusses integrating Confluent with Amazon OpenSearch using OpenSearch Ingestion.

Apache Kafka, known for its robust message tracking and retention capabilities, also accommodates the deletion of data to comply with GDPR requirements, such as the right to be forgotten. Kafka's immutable log storage presents challenges for data deletion, but the platform supports message expiration and an Admin API for explicit deletions. Businesses often utilize Kafka for long-term data storage in Event Sourcing architectures, necessitating GDPR-compliant management of enduring data. Compacted topics allow for explicit message deletion or replacement via their keys, similar to techniques used in databases like Cassandra. This can be achieved by writing a new message with a null value for deletion. Additionally, encryption per user, proposed by Daniel Lebrero, offers an alternative by rendering data unintelligible once the encryption key is deleted. Kafka's integration with databases through Connectors allows for data deletion propagation, especially when using Change Data Capture. The compaction process can be configured for continuous operation with bandwidth limitations to maintain system performance. Kafka topics can be both compacted and have expiration settings for optimal data management, with the platform's latest version, 3.8.0, bringing further enhancements and features.