May 2024 Summaries

PeerDB has released its target connector for Elasticsearch in beta. This new feature enables real-time replication from Postgres to Elasticsearch, offering benefits such as efficient search for large data volumes and denormalizing data into documents. The architecture of the connector is designed to ensure high performance and low latency during replication. PeerDB's CDC mode allows for blazing fast initial loads and constant reading of changes in Postgres through logical replication slots, resulting in sub-minute latencies.

PeerDB Streams has introduced support for row-level transformations as part of Postgres Change Data Capture (CDC). Users can write simple Lua scripts to define a transformation and add it as part of the replication. This feature allows seamless in-flight row-level transformations to Postgres data before it is streamed to the target, addressing various use cases such as masking PII data, changing data format, generated columns, unnesting JSONs, topic routing, and data encryption. PeerDB chose Lua for its balance of engineering speed, integration with their Go-based platform, and end-user usability.

PeerDB Cloud, a fully managed solution for Postgres replication, has entered public beta. It enables fast, simple, and cost-effective data replication from Postgres to various data warehouses and queues. The platform offers features such as Postgres Change Data Capture, high-quality target connectors, blazing fast parallel initial loads and resyncs, streaming query replication for production-ready replication, and a unique SQL interface for ETL. PeerDB Cloud is fully managed with zero CAPEX and OPEX costs, providing features like High Availability (HA), Horizontal Autoscaling, In-Place/Transparent Upgrades, Advanced Logs and Metrics, Privacy and Security, dedicated Slack channel, and Support SLAs. It also offers predictable pricing based on the number of vCPUs provisioned. The platform has already served over 10 production customers replicating 20TB of data from Postgres every week. PeerDB Cloud aims to contribute to making Postgres the database of the world by simplifying data movement and ETL for Postgres users.

PeerDB Streams is a new tool designed to simplify the process of replicating changes from Postgres databases to queues or message brokers such as Kafka, Redpanda, Google PubSub, and Azure Event Hubs. The tool uses native Postgres Change Data Capture (CDC) through logical decoding. PeerDB Streams aims to address the complexities of existing CDC tools like Debezium by offering a simple UI for setting up data sources and monitoring progress, an intuitive SQL layer for managing CDC, Lua scripts for row-level transformations, native connectors to non-Kafka targets, and Postgres-specific optimizations. The product is currently in beta testing and available through open source, managed service, or self-hosted enterprise offerings.

PeerDB is celebrating its one-year anniversary since joining the YC Summer 23 program with a week full of launches, starting on May 6th. The company has worked tirelessly for months to prepare for this ambitious event. To stay updated on the upcoming announcements, follow PeerDB on Twitter at @PeerDBInc.

PeerDB has introduced an efficient solution for replicating data from Postgres to ClickHouse by utilizing MinIO as an open-source alternative to Amazon S3. This integration simplifies the setup process and eliminates the need for users to provide a separate bucket for PeerDB's ClickHouse integration. The method allows for efficient replication with latencies under 30 seconds and high throughput rates. Additionally, MinIO provides a sleek UI for managing internal Avro files created during the replication process.