October 2023 Summaries

Real-time Change Data Capture (CDC) is becoming increasingly important with the growing adoption of table partitioning in PostgreSQL. PeerDB, a data movement tool for Postgres, has added extensive support for CDC for Partitioned Tables to stream real-time changes from partitioned tables to various targets like Data Warehouses, Queues, or Storage. The tool handles scenarios such as adding new partitions, dropping partitions, and managing schema changes like adding columns. PeerDB supports Postgres versions 12 to 16 and aims to improve the reliability of data movement for partitioned tables.

The text discusses a comparison between two data movement tools, PeerDB and Airbyte, specifically focusing on their performance in transferring large tables from Postgres to Snowflake. Both tools are open-source and available as Docker Compose applications. The benchmark test involved generating a 1.5TB table with 6 billion rows in Postgres and then transferring it to Snowflake using both PeerDB and Airbyte. Airbyte, which does not support parallelism, took 83 hours for the transfer. On the other hand, PeerDB implemented parallelism by logically partitioning the large table and streaming those partitions to Snowflake in parallel. With 32 threads, PeerDB completed the same task in under 5 hours, making it 16 times faster than Airbyte. Even with a single-threaded run, PeerDB was still approximately twice as fast as Airbyte. PeerDB's speed is attributed to its adoption of parallelism, configurable batching while reading from Postgres and writing to Snowflake, and the use of Avro for data transfer in binary format. The text also mentions that PeerDB is working on additional features to further improve performance. While Airbyte offers a wide variety of connectors, PeerDB focuses primarily on providing high-quality source and destination connectors for Postgres.