Launching Cloudflare's Gen 13 servers: trading cache for cores for 2x edge compute performance

Cloudflare's transition from their 12th Generation server fleet to the 13th Generation, powered by AMD's EPYC 5th Gen Turin-based processors, marked a significant leap in performance efficiency and scalability at the edge. The primary challenge was the reduction in cache size per core, which initially hindered performance due to increased latency in their legacy request handling software, FL1. By developing FL2, a Rust-based rewrite of their core request handling layer, Cloudflare successfully overcame these limitations, allowing the new system to fully exploit the higher core count and achieve linear throughput scaling without exceeding latency service level agreements (SLAs). FL2's architecture, with improved memory access patterns and reduced dependency on large caches, doubled throughput and boosted power efficiency by 50% compared to the previous generation. These advancements reduced the carbon footprint per request and enhanced Cloudflare's ability to handle global traffic efficiently, offering a compelling example of effective hardware-software co-design.