Announcing Streaming Ingest General Availability and Public Preview of File Loader in Starburst Galaxy

Starburst has announced the general availability of a fully managed streaming ingestion solution from Apache Kafka to Apache Iceberg tables, offering users a streamlined and cost-effective way to handle large-scale data ingestion at up to 100GB/second per Iceberg table. This new capability eliminates the need for complex custom software and multiple tools, providing a single, serverless solution that simplifies the process while ensuring high performance and scalability. Additionally, Starburst is introducing a public preview of file loading to further enhance data ingestion capabilities in November 2024. The platform addresses common challenges such as data scale, operational scale, and commit contention by offering dynamic load coordination, transactional dead-letter queues, and a custom commit coordination service. It also includes automated data maintenance features like compaction, snapshot expiration, and data retention to optimize Iceberg tables for performance and compliance. With these advancements, Starburst Galaxy empowers organizations to perform near real-time analytics efficiently, making it a competitive solution for businesses dealing with extensive streaming data.