Company
Date Published
Author
Lucia Cerchie, Duncan Pauly, Josep Prat
Word count
1748
Language
English
Hacker News points
None

Summary

JustOne Database has announced a new open-source sink connector that enables streaming messages at Apache Kafka speed into PostgreSQL-compatible databases, including JustOneDB, which is optimized for agile analytics and high-volume data ingestion without the need for additional indexing. The connector is designed to work seamlessly with databases supporting PostgreSQL 9.3 and above, utilizing the COPY interface for efficient data transfer, and supports synchronized delivery to ensure one-time message delivery even after restarts. The architecture capitalizes on Kafka's ability to stream vast amounts of data rapidly, while JustOneDB offers agile querying capabilities, making it ideal for building a relational data lake that facilitates data exploration and discovery across an enterprise. The connector is capable of streaming hundreds of thousands of messages per second, with performance comparable to un-indexed PostgreSQL tables, and it supports various JSON parsing configurations to convert Kafka messages into relational rows. Users can download a free version of JustOneDB to experiment with this setup, which promises fast data ingestion and analysis without the complexity of traditional big data solutions like Hadoop.