How We Export Postgres Logs to Third-Party Services

Neon has introduced the ability to export Postgres logs to any OpenTelemetry (OTLP)-compatible observability platform, addressing previously unmet user needs for direct access to log data alongside metrics. The company initially faced challenges in extending their existing metric export pipeline to logs due to limitations in the OpenTelemetry Collector's ability to scrape logs from Kubernetes pods by labels and the complexity of filtering log data from multiple processes running on NeonVM. To overcome these hurdles, Neon implemented a solution using rsyslogd to forward logs to the OpenTelemetry Collector in a structured format, enabling enrichment with Kubernetes metadata before exporting to platforms like Datadog. The implementation encountered obstacles, such as the inability to dynamically reconfigure rsyslogd without a restart and issues with matching logs to Kubernetes pods, necessitating additional steps to transform and enrich log data. The new logging capability not only enhances user experience by providing real-time access to logs but also supports compliance requirements, such as HIPAA, by enabling secure storage of Postgres logs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.