November 2023 Summaries

PeerDB has introduced the Column Exclusion feature for optimizing Postgres replication. This feature allows users to pick specific columns while replicating tables from Postgres, improving data security and privacy by excluding sensitive information. It also helps in data minimization, performance improvement, and cost savings by reducing unnecessary data transfer. PeerDB supports this feature across various targets such as Data Warehouses, Queues, and Storage. Users can exclude columns through the UI or SQL Layer while creating MIRRORs (replications).

The blog discusses the native replication of PostGIS geospatial data from Postgres to other data stores. It highlights that existing data movement tools are not reliable in replicating geospatial data, often resulting in suboptimal data type conversion and data loss. PeerDB is building a specialized data-movement tool for Postgres with extensive geospatial support, including the replication of GEOMETRY and GEOGRAPHY types to Snowflake. The blog demonstrates how to insert points, lines, polygons, and GEOMETRYCOLLECTION into a Postgres table and provides error handling for invalid geometries during replication. PeerDB plans to expand this support to other data stores such as BigQuery and Clickhouse in the future.

PeerDB has transitioned from a pull-and-push model to a more efficient streaming approach using Go channels for its Postgres-first data-movement platform. The new streaming approach involves buffering and concurrently pushing data to the target in batches, as it is pulled from PostgreSQL. This pipelining of data transfer offers significant advantages such as improved efficiency, reduced latency, efficient resource management, and better error handling. PeerDB's performance improvement is remarkable with throughput reaching 10-12k Transactions Per Second (TPS) and minimal lag of 1-5 seconds compared to roughly 30 seconds in the previous approach. Future enhancements include spilling the record stream to disk to prevent Out-of-Memory (OOM) issues for handling larger datasets without sacrificing performance or reliability.