Real-time analytics at scale: Redpanda and Snowflake Streaming

Redpanda and Snowflake combined to create a high-performance streaming data pipeline that can process 3.8 billion messages at a rate of 14.5 GB per second, achieving near real-time analytics with a P50 latency of under two seconds and a P99 latency under eight seconds. The setup was executed swiftly, transitioning from concept to production within a day, largely thanks to Redpanda's automation tools. The benchmark utilized a 9-node Redpanda Enterprise cluster on AWS EC2 instances and 12 Redpanda Connect nodes, employing the Kafka-compatible Redpanda platform and a snowflake_streaming connector optimized for high throughput and low latency. Key optimizations included using a binary format like AVRO for a 20% throughput improvement and adjusting for a delicate balance of throughput and latency. Although Snowflake's build steps introduced some latency, increasing build_parallelism and optimizing Snowpipe Streaming channels helped mitigate this. The tests demonstrated that Redpanda and Snowflake can effectively support real-time analytics in various applications, such as market surveillance and fraud detection, with insights delivered in seconds rather than hours.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.