July 2025 Summaries

Blacksmith replaced its fork-based GitHub Actions cache alternative with a transparent, colocated caching system that requires no workflow changes and avoids the maintenance burden of modified actions. After GitHub moved its cache service to a Twirp- and Azure Blob Storage-based design, the team reverse engineered its protocols using network inspection, Protobuf definitions, and assistance from an LLM, then redirected cache traffic from ephemeral Firecracker VMs through in-VM NGINX and host-level proxies to a MinIO-backed S3-compatible store. Addressing Azure SDK hostname-dependent concurrency optimizations required Azure-like URLs, DNS remapping, and proxy translation, while nftables replaced unreliable iptables rules for per-VM traffic management. The Go-based proxy streams data efficiently and uses connection pooling, and the system treats degraded cache retrieval as a cache miss to prevent slow builds. Beta testing exposed compatibility differences among third-party cache actions, enabling refinements that reportedly delivered cache download speeds up to ten times faster than GitHub Actions, including a 114 MB cache retrieval at roughly 327.5 MB/s without user changes.

Blacksmith built a GitHub Actions logging platform to help developers investigate CI failures by searching across logs rather than reviewing individual job outputs. It selected ClickHouse for storage because of its low operational overhead, strong compression, affordable scaling, fast aggregation performance, and ability to support joins with other observability data, avoiding the complexity of an ELK stack and message queue. To enable efficient substring searches across millions of log lines, the system uses ClickHouse’s n-gram Bloom filter indexes, materialized columns to infer log severity, and row-level TTL policies for customer-specific retention. Logs are securely streamed from Firecracker VMs through host-side daemons, batched and inserted asynchronously into ClickHouse, with retries and ingestion-time deduplication using ReplacingMergeTree settings. The platform also includes a lightweight Lucene-inspired query language supporting text, property, logical, and comparison searches that compiles to SQL, while ClickHouse aggregation queries power log trends and histograms; the service is available to existing Blacksmith customers.