Hunting orphan objects: 45% off our ClickHouse storage bill (and a near data-loss incident)

Tinybird faced significant challenges with orphaned cloud storage objects in their ClickHouse clusters, leading to inefficiencies and increased costs. Over time, they accumulated petabytes of unused data, which resulted in a 45% reduction in storage costs after cleanup. While cleaning up these orphaned objects, they almost lost real data due to incomplete metadata snapshots, which misclassified some legitimate data as deletable garbage. The recovery process was complex, requiring a reconstruction of relationships between various metadata sources and backup paths. This incident prompted improvements in their system, including better snapshot validation, enhanced orchestration, and more reliable recovery procedures. Ultimately, the experience not only reduced costs but also strengthened their operational safety and recovery protocols, as they continue to refine the garbage collection process and prevent future creation of orphaned objects.