Tinybird: A ksqlDB alternative for stateful stream processing

ksqlDB, an SQL-based streaming engine developed by Confluent, serves as a companion to Apache Kafka, offering real-time data processing and transformations with ease of use compared to Kafka Streams, thanks to its SQL abstraction. However, ksqlDB faces limitations in managing complexities such as high-cardinality aggregations and large time windows. It relies on Kafka for state storage, leading to inefficiencies in long-running aggregations and analytics tasks due to its reliance on a key-value storage modality that lacks data skipping optimizations. Tinybird emerges as a viable alternative, providing a highly scalable real-time analytics platform that overcomes ksqlDB's challenges by leveraging an OLAP database for efficient state management and complex analytics over unbounded time windows. Tinybird's architecture allows it to handle high throughput and stateful processing, reducing operational complexity and placing no additional strain on Kafka infrastructure. Despite ksqlDB's integration with Kafka, its potential obsolescence is hinted at, as Confluent's strategic focus shifts towards Apache Flink, raising questions about ksqlDB's long-term viability in the evolving data streaming ecosystem.