August 2019 Summaries

The text discusses the integration of graph capabilities into event streaming applications using Apache Kafka® and Neo4j, a native graph database. It explains that graphs are useful when relationships between items dominate an application, such as in social networks or financial fraud detection. By adding graph abilities to stream processing engines like ksqlDB, developers can more easily approach their use cases. The text provides a step-by-step guide on how to turn streams into graphs with Neo4j and the Neo4j-Streams plugin, including enriching and visualizing data, using powerful graph algorithms, and sending results back to Kafka. It also highlights the benefits of using Confluent Cloud for managing event streaming services without worrying about extra infrastructure management.

The text announces the general availability of Confluent Cloud Schema Registry, a fully managed service within Confluent Cloud that aids in the management of Avro schemas for event streaming applications based on Apache Kafka. This service simplifies the operational challenges associated with self-managing Schema Registry instances, offering production-level support and SLA, and can be deployed on AWS or GCP across various global regions. The Schema Registry allows for schema storage and retrieval, versioning, and compatibility checking, which are essential for maintaining data governance and ensuring application interoperability. The announcement highlights the significant increase in schema versions and addresses common operational mistakes through its managed service infrastructure. Additionally, the text offers a brief guide on setting up and using the Confluent Cloud Schema Registry, emphasizing its ease of use and integration capabilities with other systems like Amazon OpenSearch.

This article discusses an event-based architecture that leverages Apache Kafka® as a scalable and highly available single source of truth for businesses processing large amounts of data. It presents a system where all API calls are transformed into events and written to a Kafka topic using the tenant identifier as key and the remaining data as value. The system ensures atomicity, consistency, isolation, and durability in transactions, similar to ACID transactions in relational databases. It also discusses ways to handle duplicate requests or events, make the system idempotent, and scale horizontally by adding additional partitions to the tenant topic.

Confluent has introduced a revised Verified Integrations Program to support high-quality integrations from partners, simplifying verification requirements and streamlining the process. The program offers two tiers: Gold and Standard, with Gold indicating the tightest integration with Confluent Platform. To build connectors, partners can refer to the verification guide and checklist, which provide detailed documentation on development and testing practices. Several partners have already verified their work through this program, sharing end-to-end examples of how to use them, including Snowflake, MongoDB, Neo4j, Couchbase, Kinetica, DataStax, and Humio. The Confluent Hub provides a list of partner and Confluent-supported connectors, offering customers a validated and optimized way to enable operational dataflows to and from various enterprise-class databases and applications.

Apache Kafka 2.3 introduces several significant enhancements to Kafka Connect, focusing on improving task handling and logging to address common community frustrations. Notably, the adoption of incremental cooperative rebalancing through KIP-415 minimizes disruptions during task rebalancing by only adjusting necessary tasks, rather than halting all tasks as before. Improved logging with Mapped Diagnostic Context (MDC) provides clearer context for each task in the logs, aiding in troubleshooting. KIP-465 enhances the /connectors REST endpoint, allowing users to retrieve comprehensive information about connector states with fewer API calls. Additionally, KIP-458 introduces the ability to override consumer and producer configurations at the connector level, offering more granular control over settings, which is particularly beneficial for managing security protocols. These updates collectively streamline Kafka Connect's efficiency and usability, making it a more robust tool for data streaming integrations.

Confluent Platform 5.3 introduces Secret Protection, a feature designed to enhance security by encrypting sensitive data in configuration files, preventing unauthorized access to cleartext secrets such as passwords. This solution employs envelope encryption, using a master passphrase and cryptographic salt to generate a master encryption key, which encrypts a data encryption key, thereby securing the secrets. Even if a file is accessed, encrypted secrets remain unreadable without the master encryption key. This feature extends security capabilities for all components of the Confluent Platform, including brokers, Connect, and KSQL, allowing secure deployment in production environments. Users can generate, store, and deploy master encryption keys, update and rotate encryption keys, and integrate this security feature into their orchestration workflows. The platform’s CLI supports these operations, offering flexibility in managing encrypted secrets across various hosts while maintaining security best practices.

The Kafka Summit SF will feature keynotes from leading technologists, including Jay Kreps and Neha Narkhede. With 56 sessions across four tracks covering Core Kafka, Event-Driven Development, Stream Processing, and Use Cases, attendees can network with Kafka experts, meet their peers, and learn about the most interesting up-and-coming Kafka projects. Real-life Kafka use cases from companies like JPMorgan Chase, Uber, Ticketmaster, Stitch Fix, and Tesla will be highlighted. Hands-on training sessions are available for all skill levels, including Introductory Tutorial, Developer Learning, KSQL & Kafka Streams, Operations Training, and Advanced Kafka Optimization. Attendees can also participate in a three-hour certification bootcamp to become a Confluent Certified Developer or Confluent Certified Operator for Apache Kafka. The Expo Hall will feature 36 sponsors with new ideas, important adjacencies, and solutions specific to individual needs. A party, great food, and fun giveaways are also included in the event. Registration is open now, and attendees can use the code blog19 for a 30% discount.

The Apache Software Foundation has announced Tutorials for Apache Kafka, a new area on their website offering learning resources for event streaming. The tutorials are designed to make it easier for newcomers to learn about event streaming and provide a reference guide for experienced developers. The site features literate programming, continuous integration, and community-driven development, with the source code available on GitHub. The latest release of Apache Kafka 3.8.0 includes many new features and improvements, including a new JavaScript client that can be used to build an app that produces and consumes data from a Kafka topic.

Confluent Platform 5.3.0 introduces the ksqlDB UDF / UDAF Quickstart Maven archetype for bootstrapping custom ksqlDB functions. This archetype simplifies the process of building and deploying user-defined functions (UDFs) and user-defined aggregate functions (UDAFs). It also allows developers to convert the generated Maven project to a Gradle project with a simple command, catering to those who prefer using Gradle over Maven. The tutorial guides users through setting up the archetype, generating a new UDF/UDAF project, and deploying custom ksqlDB functions to a ksqlDB server. It also covers unit testing for both UDFs and UDAFs.

Imperva has successfully implemented Kafka Streams API for building shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in their system. They used Kafka Streams to maintain global state in a distributed system and address issues such as maintaining consistency and synchronization, avoiding bottlenecks, and preventing race conditions and unpredictability. The microservices built with Kafka Streams have been able to meet all requirements set by Imperva, including providing a RESTful CRUD API, distributing processing load, improving availability, preserving state stores, and scheduling tasks in response to incoming messages. Overall, Kafka Streams has proven to be very robust in their production environment, allowing them to shorten development times and bring uniformity to their code.