Store and search high-volume logs with ClickHouse and Datadog

As organizations deal with scaling AI and agentic workloads, they face challenges with managing log volumes, which can impact affordability and searchability. Datadog and ClickHouse have partnered to address this issue by introducing two new capabilities, allowing teams to route high-volume logs to ClickHouse via Datadog Observability Pipelines and search these logs directly from the Datadog Log Explorer without re-ingesting them. ClickHouse, a high-performance, open-source columnar database, excels in handling large-scale, high-cardinality telemetry data with its ability to perform sub-second analysis and reduce storage costs. The integration allows teams to strategically route application and infrastructure logs for real-time monitoring or long-term compliance storage efficiently. The Federated Logs feature enables seamless querying of ClickHouse-stored logs from within the Datadog Log Explorer, providing a unified search experience. These capabilities, currently in Preview, are designed to enhance log management by optimizing cost-effectiveness and maintaining searchability across extensive datasets.