November 2022 Summaries

The blog post explores how DeltaStream facilitates seamless integration with various data serialization formats such as JSON, Protocol Buffers, and Apache Avro, simplifying the process of creating real-time analytics dashboards. The example provided illustrates an eCommerce scenario where transactional data in Protobuf and customer data in Avro are converted into JSON format for a dashboard. Key metrics include real-time revenue sums and geographic transaction counts. DeltaStream handles the complexities of serialization and deserialization behind the scenes, allowing users to focus on writing SQL queries for data transformation and integration. The post also highlights the use of DeltaStream descriptors and schema registries to manage data formats effectively, demonstrating how these tools streamline the development of pipelines for real-time insights without the burden of managing complex streaming applications.

The second installment of the DeltaStream 101 series explores how DeltaStream can enrich, filter, and aggregate streaming data from different sources like Apache Kafka and Amazon Kinesis to meet product data needs more efficiently. By using a real-life example, the process demonstrates enriching a clicks stream by integrating user data from a Kinesis changelog, which allows for counting clicks per URL and device type. This enriched stream is then used to aggregate clicks per URL and region, facilitated by DeltaStream's capability to perform temporal joins and maintain updated user information. The setup involves declaring a Kinesis store and creating changelogs to manage user information updates. By leveraging SQL-based queries, developers can build complex streaming applications quickly without the need for extensive maintenance, showcasing DeltaStream’s potential to simplify and enhance stream processing.