Born Expired: How We Cut ClickHouse's S3 Write Bill by ~92%

A large write-heavy ClickHouse observability deployment adopted an S3 cold tier after local EBS volumes and merge bandwidth became capacity constraints, retaining recent, actively merging data on local disks and moving stable parts through a TTL-based storage policy. Although the architecture worked and disaster-recovery tests showed that local metadata pointers enabled fast cold-tier recovery, S3 request charges—not storage capacity—accounted for roughly 96% of costs, driven by many-object Wide parts, late-arriving telemetry immediately qualifying for event-time-based cold-tier TTL movement, cold-part rewrites, and duplicated writes across independent replicas. Testing also found that ingestion performance depended mainly on insert batch size, partition fan-out, and disk bandwidth rather than instance CPU or S3 tiering. The team reduced PUT requests by 59% using metadata-only changes that raised the Wide-part threshold to favor Compact parts and kept small, high-churn tables off S3, then projected a roughly 92% reduction by changing TTL calculations from event time to insertion time so late data could merge locally before moving once as larger parts. They retained independent replica storage for durability, rejected zero-copy replication and unavailable packed-part features, and concluded that object storage tiering is primarily a cost-and-capacity strategy whose economics require monitoring request counts, part formats, and TTL behavior rather than stored bytes alone.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.