February 2019 Summaries

A recent announcement highlights the availability of the Kafka Connect Neo4j Sink from Confluent Hub, enabling users to stream events from Apache Kafka directly into the Neo4j graph database. Neo4j, an open-source graph database, uses the Cypher query language to manage and query data represented as nodes and relationships. This integration is designed to handle event data from various sources, including frontend systems and other databases, and allows for the configuration of Cypher statements per Kafka topic to control data ingestion. Users can install Neo4j locally, via Docker, or on cloud platforms, and developers can interact with its features through multiple programming languages. The announcement also emphasizes the importance of community feedback and testing in real-world Kafka and Neo4j environments, encouraging users to provide insights and raise issues on GitHub to help improve the integration.

Serverless functions, particularly those enabled by FaaS (Function-as-a-Service), are increasingly being used in event-driven architectures, particularly with Apache Kafka. The key to integrating FaaS into an event streaming model is understanding how FaaS runtime characteristics fit into the event streaming paradigm. FaaS can be used for atomic, stateless processing, and it's well-suited where events are treated as facts, providing a cost-efficient solution. However, FaaS may not be suitable for non-stream-oriented stateful processing or when time sensitivity is essential, as it introduces latency and potential issues with stale data. The synergy between FaaS and event streaming platforms is natural, enabling rich collaboration between serverless runtimes and the event streaming platform. As the technology continues to evolve, FaaS principles are being developed to ensure seamless integration with cloud events and streaming platforms, pushing down barriers to cloud adoption and enabling greater scalability and complexity.

Spring for Apache Kafka brings the familiar Spring programming model to Kafka, providing features such as publishing records with the `KafkaTemplate` and asynchronous execution of POJO listeners through a listener container. It also includes error handling mechanisms like the `SeekToCurrentErrorHandler`, which discards remaining records from the poll() method and performs seek operations on the consumer to reset the offsets. The `DeadLetterPublishingRecoverer` can be used to send failed messages to another topic, known as a dead letter topic. Additionally, Spring Boot provides auto-configuration for much of the infrastructure, allowing developers to focus on their business logic. The framework also supports deserialization exceptions and provides tools for type mapping between different classes, making it easier to work with Kafka in a Spring-based application.

The text discusses the implementation of a flexible, managed repository for Protobuf schemas in an event streaming platform built with Apache Kafka and MongoDB. The goal was to provide strong guarantees on data quality and schema evolution while supporting service decomposition work and analytical data needs. The authors investigated various encoding formats, ultimately choosing Avro due to its support for managing schema evolution and backwards and forwards compatibility. However, they discounted Avro due to lack of cross-language support. Instead, they chose Protobuf, which was already being used in the platform. To ensure data consistency and schema evolution, the team implemented unit tests that enforce rules on message structure and field numbers. They also created a centralized repository for schemas and introduced custom options in Protobuf IDL to enforce topic metadata and schema relationships. Additionally, they developed a Producer API that performs schema/topic validation before forwarding messages to Kafka, providing a method to enforce the relationship between producers, topics, and schemas. The platform now offers strong guarantees on data quality and schema evolution while supporting service decomposition work and analytical data needs.

During a presentation titled "Hunters ATT&CKing with the Right Data" at ATT&CKcon, the importance of modeling security event logs for threat hunting was emphasized, particularly through the use of HELK, a free threat-hunting platform leveraging the Elastic stack, Apache Kafka, and Apache Spark. The discussion highlighted how KSQL, a SQL-like query language for Kafka Streams, can facilitate real-time security detection by joining Windows Sysmon event logs to detect suspicious activities such as lateral movement techniques. By streaming and processing these events in real-time, KSQL enables the creation of enriched data streams, offering a more efficient alternative to traditional security information and event management (SIEM) systems that often rely on query-time processing. The blog post further explores the benefits of using KSQL for joining data streams and underscores the potential for real-time analysis and alerting, which is crucial for effective cybersecurity operations.

The Confluent team is planning three Kafka Summits in 2019, with each event having its own unique news and agenda. The first Summit will be held in San Francisco and will feature a large number of speakers and attendees, while the second Summit will take place in London with an equally impressive lineup of sessions and keynote speakers. The third Summit will be held in New York, which is expected to sell out quickly due to its popularity. Additionally, Confluent is hosting training events in conjunction with each Summit, providing developers with opportunities to learn more about Apache Kafka and event streaming architectures. The team is also promoting the Current 2024 conference in Austin, Texas, where industry leaders will gather to discuss the latest developments in data streaming and cloud services.

The event-driven architecture paradigm shift leads to clarity and inspiration, and designing systems for payment processing at scale using this approach is crucial. Event-driven programming models have evolved over the years, from distributed object (RPC sync), service-oriented architecture (SOA), enterprise service bus (ESB), to reactive programming and microservices. The event-streaming platform provides a data platform that behaves accordingly, with data models developed in relation to use cases. Stream processors are used for constructing flows from other streams, enabling logic execution or splitting, merging, and maintaining materialized views. Persistence is significantly different in the streaming model, where events are front and center, and state is maintained by event processors using incoming streams. The event streaming platform provides elasticity, correctness guarantees, and solves conventional microservice concerns. Event sourcing is another mechanism that can use a specific framework or be built into various implementations, but requires dedicated process control to ensure point-in-time or snapshot state is correctly rebuilt. Stream processor stateful operations like join, enrich, window, aggregate, etc., are stored at the partition level, and all of these elements compound upon each other with the repeated notion that events capture facts, streams capture behavior, and a series of related events tells a story. The event streaming platform replaces microservice frameworks and solves first-class concerns like event routing, service discovery, persistence, and elasticity, enabling continuous extension of functionality and evolutionary change.

The integration of Apache Kafka with relational databases is popular due to their rich source of events, allowing existing data and changes to be streamed into Kafka topics. This enables data to drive applications, be streamed to other stores, and stored for analytics. The JDBC connector for Kafka Connect, available with Confluent Platform or separately from Confluent Hub, facilitates this process by pulling data from databases like Oracle, SQL Server, MySQL, and Postgres into Kafka and vice versa. Users can configure Kafka Connect to ingest data from databases by ensuring the JDBC driver is accessible and setting up appropriate configurations, such as connection URLs and table selections. Options include bulk data import or incremental data streaming based on column changes, with Single Message Transforms (SMTs) enabling further customization of Kafka messages. Concurrency can be managed by adjusting the number of tasks to optimize database connections. The article provides detailed instructions and troubleshooting tips for setting up and managing these configurations, emphasizing the importance of correct driver installation and error handling.

Building a scalable and reliable machine learning infrastructure is a complex task that extends beyond simply creating analytic models with Python. The blog post discusses the challenges of integrating various components, highlighting Uber's Michelangelo platform, which initially relied on Apache Spark and Java but later expanded to support Python models and frameworks like PyTorch and TensorFlow. The Apache Kafka ecosystem is presented as a solution to the impedance mismatch often faced between data scientists, data engineers, and production engineers, offering a scalable and reliable system for data ingestion, processing, and model deployment. By integrating Kafka with tools like KSQL and Python environments such as Jupyter Notebooks, data scientists can perform interactive data analysis and preprocessing while benefiting from Kafka's scalability and reliability for production deployment. The post emphasizes the importance of resolving these integration challenges to unlock real business value from machine learning projects and suggests that Kafka and Python are complementary technologies that can be effectively combined to support various machine learning workflows.

Zenreach, a company focused on enhancing customer engagement for brick-and-mortar merchants, transitioned from a Python script to Kafka Streams to improve the scalability and reliability of their walkthrough count system, which tracks the number of walk-ins generated through Zenreach Messages. The previous system, reliant on Cassandra and MongoDB, struggled under load and added read pressure to these databases, prompting the move to a streaming solution. Kafka Streams enabled Zenreach to implement a distributed system that processes Zenreach Messages and walk-in data into walkthrough counts using services like WalkthroughsGenerator and WalkthroughsMongoExporter, alleviating the need for additional infrastructure. The switch to Kafka Streams presented challenges such as handling out-of-order data and testing punctuate logic, leading to insights on effective use of Kafka's features like TopologyTestDriver for testing and the necessity of correct partitioning in state stores. The author also shared lessons learned and tips for designing robust Kafka Streams applications, emphasizing the importance of understanding Kafka Streams architecture for proper data processing and partition management.