Cloud Topics: Level Zero garbage collection

Redpanda's approach to garbage collection in its Cloud Topics involves carefully managing Level Zero (L0) objects, which are temporary data batches, and transitioning them to Level One (L1) objects for historical reads. The goal is to efficiently determine when L0 objects are no longer needed and can be safely deleted without manual intervention. Redpanda utilizes a system of epochs, which act as logical timestamps embedded in L0 object IDs, to track the progression and safe deletion of these objects. Through per-partition epoch tracking embedded in the Raft log, Redpanda achieves a distributed system where epochs are updated monotonically, ensuring that once an epoch is deemed safe for garbage collection, it remains so permanently. The system allows for flexibility in handling partition leadership changes and reduces the risk of rejecting new writes due to stale epoch caches. This design ensures effective garbage collection while maintaining system safety and efficiency, with a promise of continued improvements in future updates.