December 2023 Summaries

In 2023, DeltaStream experienced a transformative year marked by significant advancements in making stream processing accessible and powerful for all users. The company expanded its expertise by establishing a Go-To-Market team, enhancing community engagement through educational content such as blog posts and webinars. It launched a public SaaS offering and a private "Bring Your Own Cloud" (BYOC) solution, the latter ensuring data sovereignty by running the data plane within the user's VPC. DeltaStream deepened its ecosystem integration by enabling seamless connections with Snowflake and Databricks and open-sourced its Snowflake connector for Apache Flink, fostering collaboration. Additionally, the company introduced Change Data Capture (CDC) for PostgreSQL, offered a free trial to showcase its features, and achieved SOC 2 Type II compliance, highlighting its commitment to data security. DeltaStream actively engaged with the community by participating in key conferences and events, solidifying its position as a leader in stream processing solutions and setting the stage for future growth and innovation.

DeltaStream emphasizes robust data security and privacy by implementing a zero trust strategy, where security verifications exist at multiple levels to minimize exposure in case of breaches. The platform ensures queries for data processing are run in isolated environments using Apache Flink, preventing unauthorized access and resource competition. DeltaStream stores only necessary data, securely encrypting any data at rest or in transit. Users can choose between Public, Dedicated, or Bring Your Own Cloud (BYOC) deployment models, allowing them to select the best option for their security needs. Additionally, DeltaStream offers federated data governance by combining data federation and governance, simplifying access control through a hierarchical namespace model and Role-Based Access Control (RBAC) for managing permissions across disparate data sources. This approach helps organizations share data securely on a need-to-know basis, reducing management overhead and potential errors.

Stream processing is a critical tool in cybersecurity, allowing for immediate action on potential threats by processing data with minimal latency. DeltaStream exemplifies this capability by enabling the detection of suspicious user login activities through a streaming analytics pipeline. By leveraging a Kafka Store topic for failed login events, a continuous query identifies users attempting to access multiple accounts within a short timeframe, flagging them as suspicious. This rapid detection facilitates immediate follow-up actions such as account freezing and notification alerts. The system's SQL-based approach supports not only this simple use case but also more complex scenarios, highlighting DeltaStream's versatility in managing and securing streaming data.

Pattern recognition in data processing, particularly for detecting anomalies and sequences, has been enhanced in SQL with the introduction of the MATCH_RECOGNIZE syntax, part of the SQL standard since 2016. This syntax allows users to write efficient SQL queries to address pattern recognition problems, applicable in both batch and real-time contexts. A blog post from DeltaStream explores using MATCH_RECOGNIZE within the DeltaStream SQL syntax to identify bus trips with significantly increasing delays, offering a detailed breakdown of a sample query. The process involves creating a stream with projected fields, partitioning and ordering data by specific criteria, and defining patterns with variables (A, B, C) for which conditions must be satisfied, such as increasing delays. The MATCH_RECOGNIZE syntax, while initially complex, is demonstrated to be a powerful tool for real-time data analysis and pattern detection.