Refactoring a SQL Table at Scale: Lessons from Harness CI

Harness refactored its Continuous Integration Test Intelligence service after a flat, denormalized test-result table became inefficient at millions of records, causing repeated storage of hierarchy strings, slow text-based queries, full-scan summaries, payload-dependent API latency, and memory pressure. The new design separates fast API acceptance from asynchronous processing, streams uploads into storage with fixed memory overhead, uses normalized tables with integer foreign keys, incrementally merges report deltas, and maintains pre-aggregated counters for constant-time summary retrieval. It also uses hybrid storage, retaining small reports as compressed database blobs and placing large reports in columnar object-storage files queried through an embedded analytical engine, while workers scale horizontally through queues and distributed coordination. Load testing exposed further bottlenecks in indexed database writes, ID lookups, and autoscaling speed, leading to staging tables, batch-oriented lookup patterns, and more responsive scaling. The migration preserved existing API contracts and used feature flags to write to both systems before progressively shifting reads, while AI-assisted design iterations helped examine edge cases alongside planned unit and integration testing.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.