Clever Cloud Offloading 20TB of Infrastructure Data Every Month with Kestra

Clever Cloud, a European Platform as a Service (PaaS) provider, has automated its data archiving process using Kestra to manage an influx of monitoring data that reaches 20TB monthly. This robust solution addresses challenges in data management by utilizing Warp10 on FoundationDB to efficiently handle and compress large volumes of data, facilitating resource allocation that ensures optimal application performance. The automation process orchestrated by Kestra involves a structured workflow that manages data fetching, compression, and offloading to long-term storage while maintaining system performance through parallel processing and error handling. The successful integration of Kestra has significantly improved data volume management, storage optimization, and system performance, encouraging further automation efforts within Clever Cloud's infrastructure.