Company
Date Published
Author
Tyler Rockwood
Word count
1426
Language
English
Hacker News points
None

Summary

Redpanda has introduced a new Snowflake Connector in Redpanda Connect, available across major cloud providers like AWS, GCP, and Microsoft Azure, aiming to enhance data streaming and integration for real-time analytics. The connector offers up to twice the throughput on the same hardware compared to Kafka Connect, facilitating efficient and flexible data streaming into Snowflake's Snowpipe Streaming feature. Snowpipe Streaming allows high-throughput data ingestion with exactly-once semantics and low latency, supporting various use cases such as IoT data ingestion and log ingestion into ML feature stores. The collaboration with Snowflake led to the redesign of their Java SDK into Golang, improving performance with parallelized file construction and optimized assembly. Benchmarks showcase Redpanda Connect's efficiency, achieving significantly higher throughput than Kafka Connect. This development is expected to streamline data-intensive applications on Snowflake, with Redpanda Connect supporting various input sources and facilitating schema management through simple configuration. The previous Snowflake connector, snowflake_put, will be deprecated, urging users to transition to snowflake_streaming.