October 2025 Summaries

Cloudflare's exploration of "BGP zombies"—routes that become unexpectedly stuck in the Internet's Default-Free Zone due to missed or lost prefix withdrawals—delves into the complexity of how these zombies form and impact network operations. These zombies can lead to routing inefficiencies, like trapping packets in loops or causing them to take unnecessarily long paths, thus affecting network performance. The article explains the concept of path hunting, where routers search for the best path to a prefix, and how this process can lead to zombie creation, particularly when more-specific BGP prefixes are withdrawn. Cloudflare highlights the role of BGP path hunting and Minimum Route Advertisement Interval (MRAI) in exacerbating these issues, and details how zombies can occur between upstream ISPs or within a network, causing traffic disruptions. To mitigate these problems, Cloudflare suggests improvements such as better traffic forwarding and a multi-step draining process for customers using on-demand BGP functionality. They also advocate for the implementation of the BGP SendHoldTimer to combat routing bugs that lead to zombie creation. Cloudflare's efforts aim to enhance the stability and efficiency of Internet traffic management, ultimately contributing to a more reliable global network.

In September 2025, a Cloudflare engineering team faced a critical error related to two internal microservices struggling to communicate due to an HTTP/2 mitigation issue, specifically a PING flood attack. The blog post explores the complexities of HTTP/2, a protocol with powerful features that can be easily misused, leading to denial-of-service risks. Cloudflare's defenses, including the use of the error code ENHANCE_YOUR_CALM, aim to mitigate these risks by closing potentially malicious connections. The team discovered that excessive PINGs and RST_STREAM frames were being sent due to a bug in Go's standard library, highlighting the importance of reading response bodies in full to avoid unnecessary frame transmissions. The incident underscored the benefits of using Cloudflare's own infrastructure for internal services, allowing the team to identify and resolve such issues, thereby enhancing system security and reliability for both internal and external users.

The Internet is evolving with the increasing use of AI agents, which can perform tasks like ordering pizza, buying tickets, and more, by interacting with websites on behalf of users. This shift is expected to change traffic patterns significantly, with more requests coming from AI platforms rather than traditional devices. As this transition occurs, there are challenges in ensuring network security and user privacy, as AI platforms can become sources of cyberattacks. Existing security tools may not be sufficient to handle these changes, prompting the development of more refined mechanisms, such as anonymous credentials (AC), which allow websites to enforce security policies without identifying users. Cloudflare is exploring the use of ACs, which are under development at IETF, to maintain Internet security and privacy in this new AI era. These credentials offer a way to rate-limit users and block malicious activity without compromising user anonymity, promising a crucial role in the future of Internet interactions. The company is contributing to this development and encourages others in the space to participate, with ongoing efforts to refine these technologies for real-world applications.

As the need for cryptographic signing of requests by bots and agents increases, website operators face challenges in discovering public keys for verification, particularly for lesser-known entities. This issue, known as discovery, is shared by platforms like Amazon Bedrock AgentCore, which aims to facilitate individual agent signatures rather than a monolithic approach. Cloudflare proposes a registry of bots and agents to simplify key discovery, akin to existing IP and robots.txt lists. This registry would allow operators to configure traffic permissions while fostering an open curation ecosystem. The Web Bot Auth protocol, proposed in May, has seen multiple implementations and aims to shift from brittle identification methods to more reliable cryptographic authentication. To support this, a registry format with a list of URLs for agent keys and a signature-agent card format providing metadata like operator contact and expected crawl rate are proposed. Cloudflare plans to provide one of the first registry instances, encouraging contributions and references from other operators.

The Internet is undergoing a significant transition towards post-quantum (PQ) cryptography to safeguard against potential quantum attacks, with particular focus on transitioning systems like TLS connections to PQ alternatives. While PQ cryptographic algorithms have increased costs, making them challenging to integrate seamlessly, efforts are progressing, such as Cloudflare's work on TLS and Anonymous Credentials (ACs), which address privacy issues by allowing users to prove specific attributes without revealing excessive personal information. The European Union's upcoming digital identity wallet and Cloudflare's rate-limiting applications are examples of real-world AC deployments, yet the migration to PQ alternatives remains complex due to the need for new cryptographic primitives, such as lattices or hash functions, that can ensure security against quantum threats. The blog discusses various PQ techniques, including zero-knowledge proofs and multi-party computations, but highlights the lack of efficient, standardized zero-knowledge-friendly hash functions as a significant barrier. The article calls for industry and academic collaboration to develop robust post-quantum protocols that address practical needs, such as per-origin rate-limiting and stateful credentials, to prepare for a post-quantum world.

On September 28, 2025, Moldova held a crucial parliamentary election amidst concerns of Russian interference and significant cyber threats. The election was seen as a pivotal moment for the country's geopolitical direction, with a pro-European government competing against a pro-Russian opposition. To ensure the integrity of the democratic process, Cloudflare supported the Moldovan Central Election Commission (CEC) by deploying cybersecurity measures to counteract a series of Distributed Denial of Service (DDoS) attacks targeting election-related websites. These attacks, which peaked at over 324,333 requests per second, were part of a broader campaign against official institutions and public information channels, but Cloudflare's automated defenses successfully mitigated them, maintaining the availability and integrity of the electoral services. The Moldovan government confirmed the attacks and expressed gratitude for the protection provided, which allowed citizens to access real-time election results without disruption, ultimately leading to a victory for the pro-Western government.

Cloudflare leverages Linux's networking capabilities, particularly its Netfilter subsystem, to handle billions of requests while mitigating DDoS attacks, and this adaptability was crucial in developing WARP, a mobile-first performance and security app. Facing the challenge of securely and efficiently egressing user packets from edge machines, Cloudflare initially implemented a high-performance VPN using Linux's capabilities, integrating it into their existing network and employing techniques like network address translation (NAT) and conntrack to manage packet flow and security. This setup allowed Cloudflare to route and firewall WARP traffic effectively, though the IPv4 address exhaustion posed scalability challenges, prompting considerations for IP sharing to avoid high leasing costs. The exploration of these solutions and the innovative use of Linux's networking features illustrate Cloudflare's continuous adaptation and development in response to technical challenges, with further insights expected in subsequent explorations.

IP addresses, traditionally used as stable identifiers for geolocation and security, face challenges due to internet changes like Carrier-Grade Network Address Translation (CGNAT), VPNs, and proxies, which result in a single IP representing numerous users. This shift can cause collateral damage when security measures, based on outdated assumptions, unfairly affect users, particularly in developing regions with fewer available IP addresses. Cloudflare addresses these issues by developing a method to detect large-scale IP sharing, distinguishing CGNAT from other IP-sharing technologies using machine learning models trained on a comprehensive dataset. This approach aims to minimize biases and improve network operations and digital equity, considering the socio-economic and geographical variations in IP usage. Cloudflare's efforts focus on enhancing their security measures and assisting customers in configuring better protections, while inviting collaboration from ISPs operating CGNAT to foster a more equitable internet experience.

Internet connections, fundamental to all online activity, are composed of streams of data packets between devices. Although local data capture is possible, global measurement remains challenging due to scale and access constraints. Cloudflare's study of TCP connections provides insights into connection characteristics through their global CDN, with data showing that 70% of HTTP requests are TCP-based. The study reveals the heavy-tailed nature of internet traffic, where a small portion of connections handle large data volumes, while most involve minimal data transfer. Key metrics such as packet counts, data volume, connection duration, and request counts highlight variations between different HTTP protocols and the asymmetric nature of data exchange. The data also offers insights into path characteristics like Path MTU and initial congestion windows, crucial for understanding network efficiency and performance. This research aims to enhance network performance and reliability, with findings suggesting potential improvements in initial congestion window sizes, which could significantly affect connection performance. By publishing their findings, Cloudflare seeks to promote collaboration and further improvements in internet connectivity and reliability.

Cloudflare's exploration of the Linux networking stack reveals challenges and solutions in optimizing network performance, particularly through their "soft-unicast" method of sharing IP addresses across data centers. The company developed a service called SLATFATF, or "fish," to manage the egress of IP packets using soft-unicast address space, which is crucial for products like WARP. This service addresses issues encountered with Linux's socket subsystem and Netfilter conntrack module, such as port collisions and inefficiencies in packet forwarding. By exploring innovative solutions, including the use of TCP_REPAIR and TCP Fast Open, Cloudflare aims to streamline network operations while maintaining high performance. Ultimately, the company found the benefits of terminating TCP connections internally outweighed the complexity of supporting both IP packet forwarding and socket usage. This journey underscores the complexity of scaling network solutions and the need for creative engineering approaches, inviting those interested in tackling such challenges to join their team.

In April 2025, Cloudflare addressed two security vulnerabilities (CVE-2025-4820 and CVE-2025-4821) within their open-source QUIC protocol implementation, quiche, which were identified through their Public Bug Bounty program. These vulnerabilities involved DDoS risks related to packet acknowledgement (ACK) handling, allowing potential attackers to exploit lack of ACK validation to artificially inflate send rates and gain unfair network advantages. The vulnerabilities, though not exploited, were promptly patched by Cloudflare, enhancing ACK range validation and implementing dynamic congestion window (CWND)-aware skip frequency to mitigate attacks like the Optimistic ACK attack. This approach ensures fairness and prevents malicious clients from leveraging network resources excessively. Cloudflare's proactive response also involved collaboration with researchers who disclosed the vulnerabilities, enabling the bolstering of security across multiple QUIC implementations.

As the global race to develop quantum computers intensifies, concerns arise over the potential threat these machines pose to current cryptographic systems, which secure much of the internet's data. To address this, Cloudflare is aiding the transition to Post-Quantum (PQ) cryptography, safeguarding about half of its network traffic against future decryption threats posed by quantum computers. However, adopting PQ cryptographic methods, such as those needed for TLS certificates, presents challenges due to their significantly larger size and associated performance issues. Merkle Tree Certificates (MTCs) offer a promising solution by minimizing the number of signatures and public keys required during the TLS handshake, thus mitigating performance drawbacks. In collaboration with industry partners, Cloudflare plans to experimentally deploy MTCs, working with Chrome Security to test their effectiveness without compromising current security standards. This effort is part of a broader initiative to ensure the internet remains secure and private, even as quantum computing capabilities advance.

By the end of October 2025, a significant milestone in internet security was reached with the majority of human-initiated traffic with Cloudflare using post-quantum encryption to counter the potential threat posed by quantum computers. This progress marks an ongoing transition toward post-quantum cryptography, which aims to safeguard against quantum computers that excel in solving specific problems, like breaking widely used cryptography such as RSA and elliptic curves. While current quantum computers are not yet capable of decrypting existing cryptography, the risk of "harvest-now-decrypt-later" attacks necessitates a migration to post-quantum standards. Advances in quantum hardware, such as Google's achievement of a scalable logical qubit, and software optimizations, notably Craig Gidney's work reducing the required qubits to crack RSA-2048, have brought this migration into sharper focus. Despite the challenges of adapting protocols like TLS for post-quantum cryptography, progress is being made, with over half of internet traffic now protected by post-quantum key agreements. However, migrating to post-quantum signatures presents more complexity due to larger data sizes and the need for new standards, prompting ongoing exploration of efficient solutions like Merkle Tree Certificates. While regulatory timelines are set for 2030-2035, the path to a fully quantum-secure internet remains gradual, requiring vigilance and adaptation from organizations worldwide.

On July 8, 2022, a massive outage at Rogers, a major Canadian telecom provider, highlighted the critical importance of Internet resilience, which refers to a network's ability to withstand, adapt to, and recover from disruptions. Unlike mere reliability or robustness, resilience encompasses diverse and secure routing paths, rapid connectivity restoration, and the collective interaction of thousands of autonomous networks. This resilience is built upon diverse physical infrastructure, secure routing practices, and competitive market structures. Local decisions in network configurations significantly impact global connectivity, with the Internet's decentralized structure being both a strength and a challenge for resilience measurement. The document outlines a data-driven framework for assessing Internet resilience using publicly available data, focusing on metrics such as interconnection patterns, route hygiene, and economic-weighted metrics. The framework emphasizes the role of practices like RPKI, ROV, multi-homing, and participation in Internet Exchange Points (IXPs) to enhance resilience. Additionally, it discusses the impact of submarine cables and inter-domain routing on resilience, advocating for the implementation of best practices and public measurement platforms to foster a more stable and secure Internet.

Cloudflare has introduced an Open Beta for tracing on its Workers platform, aimed at enhancing application visibility and performance diagnostics without requiring manual setup or code changes. This automatic instrumentation captures OpenTelemetry-compliant spans, providing detailed metadata and timing information for every operation performed by a Worker, thus helping developers identify performance bottlenecks and resolve errors. The tracing feature, available in the Cloudflare dashboard, allows developers to view traces, query across Workers, and export data to OpenTelemetry-compatible providers for integration with existing observability stacks. Cloudflare plans to expand this feature by adding more automatic traces, custom spans, and the ability to export metrics to third-party providers, with pricing set to begin on January 15, 2026, for both viewing and exporting traces.

Throughout the third quarter, a multitude of Internet disruptions were observed globally, stemming from various causes including government-directed shutdowns, technical issues, and natural disasters. These disruptions often coincided with significant events such as exams in countries like Syria, Iraq, and Sudan, where authorities curtailed Internet access to prevent cheating. Physical disruptions also played a role, with incidents like cable damage due to construction and unusual mishaps such as a stray bullet in Texas. In addition, cyberattacks and technical failures further contributed to connectivity issues, exemplified by targeted attacks in Yemen and outages in Iran and Pakistan. The report highlights that while some disruptions were acknowledged by service providers, others remained unexplained, emphasizing the complexity of maintaining stable Internet connectivity in the face of diverse challenges. Cloudflare's Radar platform continues to monitor and report these events, offering insights into the underlying causes and impacts of Internet traffic anomalies.

Cloudflare Radar, launched in 2020, provides a comprehensive overview of Internet patterns, focusing on security, performance, and usage through aggregated data from Cloudflare services. It offers insights into Internet health by tracking aspects like security threats, protocol adoption, and AI bot activity, while continuously expanding its data sets and capabilities. Recent updates have enhanced its transparency and usability, such as the Certificate Transparency section launched in 2025 and the addition of AI Insights, which addresses the impact of AI crawlers on content creators. Radar also supports Internet resilience with advanced routing visibility features, including real-time BGP route visibility and AS-SET monitoring. The platform's API and MCP server enable programmatic access to its data, facilitating integration into users' tools and applications. Cloudflare Radar remains committed to evolving its tools and data sets to provide actionable insights into the constantly changing Internet landscape.

In October 2025, Cloudflare launched a new TLD page on Radar, utilizing aggregated data to provide insights into the popularity, activity, and security of top-level domains (TLDs). This initiative is part of a broader effort to enhance understanding of TLD dynamics by offering detailed metrics such as DNS Magnitude, which estimates a TLD's visibility based on unique client queries. The project covers various TLD types including generic, country code, and sponsored TLDs, and provides individual pages with detailed information for each delegated TLD, including DNS query volume and certificate issuance data. The .su TLD, historically assigned to the Soviet Union, surprisingly tops the DNS Magnitude ranking due to its wide query distribution, although its use is primarily driven by a popular online game. The new page aims to offer valuable insights for TLD managers and site owners, and plans for 2026 include incorporating performance data of name servers to assist in domain registration decisions. The data is available through Cloudflare's API, with tools for interactive exploration and sharing on social media platforms.

Measuring the Internet is a complex yet essential scientific practice due to the network's opaque nature, which stems from its composition of independently operated and rarely transparent systems. Internet measurement requires rigorous methodologies, ethical considerations, and effective data representation to avoid misleading conclusions and unintended consequences. The measurement process involves data curation, modeling, and validation, each contributing to understanding Internet behavior and facilitating accurate predictions. Cloudflare exemplifies the importance of integrating passive and active measurement techniques, as shown in their analysis of HTTP traffic spikes in Ukraine, which underscored the necessity of context and alternative explanations. Visualization plays a crucial role in illustrating data insights while mitigating biases, and ethical considerations often drive innovative methodologies, such as Cloudflare's approach to estimating CDN performance without intrusive techniques. The interplay of these elements ensures the effectiveness and reliability of Internet measurement, fostering a deep understanding of the global network.

Cloudflare aims to hire 1,111 interns by 2026, providing numerous opportunities for developing and implementing code, as well as measuring complex Internet phenomena. Despite the vast data available at Cloudflare, analyzing it to detect Internet tampering by network middleboxes is challenging due to the scale and noise in the data. Former intern Ram Sundara Raman, now an assistant professor, leveraged his experience in active network measurements to develop methods for detecting such interference using Cloudflare's passive data. His internship focused on identifying tampering patterns without active probing, revealing the complexity of interpreting large-scale data. The project underscored that large infrastructure providers, despite extensive data, face difficulties in drawing clear conclusions about Internet tampering. Raman suggests that a combination of passive and active methods could enhance detection capabilities, and he continues to explore this at UC Santa Cruz. This collaboration highlights the potential for academia and industry to contribute to a better Internet, inviting more prospective interns to join such initiatives.

Cloudflare Research is actively exploring new technologies to enhance the Internet, engaging in collaborative efforts with academia, standards bodies, and the open-source community to test hypotheses and implement ideas in real-time. This week, they are publishing a series of blog posts coinciding with the ACM's Internet Measurement Conference, focusing on Internet measurement, resilience, cryptographic protocols, and advances in networking technologies. The posts aim to clarify Internet measurement complexities, outline frameworks for Internet resilience against disruptions like cyberattacks or natural disasters, and discuss progress in post-quantum Internet technologies, including Merkle Tree Certificates. Additionally, Cloudflare will offer insights into their network's operation, the challenges of scaling IP characterization, and innovations in Linux networking for improved performance. They will also address the impact of AI on infrastructure and propose cryptographic protocols to ensure security and resilience in an increasingly automated web environment.

Cloudflare's Speed Test offers a unique approach to evaluating Internet quality by focusing on realistic usage patterns rather than just maximum throughput. Unlike traditional speed tests that aim to saturate a connection to measure peak bandwidth, Cloudflare assesses connection quality through data blocks of predefined sizes, taking into account factors like latency, jitter, and packet loss, which are crucial for tasks such as video conferencing and gaming. Operating on Cloudflare's global network, the test provides an Aggregated Internet Measurement (AIM) score to translate performance into user-friendly terms. It accounts for variables beyond the ISP's control, such as local Wi-Fi conditions, making it a comprehensive tool for understanding overall connectivity. Cloudflare's Speed Test is open-source and contributes anonymized data to public datasets, supporting open Internet research. Future enhancements aim to expand its capabilities for more detailed diagnostics and to empower third-party integrations, with the goal of improving network performance for a wide range of stakeholders.

The emergence of agentic commerce, where AI agents conduct transactions on behalf of consumers, presents new security challenges, prompting Cloudflare to partner with Visa and Mastercard to secure these automated processes. Visa's Trusted Agent Protocol and Mastercard's Agent Pay help merchants differentiate between legitimate AI agents and malicious bots through the use of Web Bot Auth, which employs HTTP Message Signatures with public key cryptography for agent authentication. This collaboration ensures agents provide verifiable, time-based, and non-replayable requests, allowing merchants to manage interactions during both browsing and payment phases. Cloudflare acts as a validator, helping merchants by confirming the authenticity of agent interactions through cryptographic checks and aiding in the seamless integration of these protocols without infrastructure changes. The initiative also involves developing support for these protocols within Cloudflare's Agent SDK and managed rulesets, facilitating secure and authorized automated commerce while maintaining site security and enabling revenue opportunities through trusted agents.

In September 2025, attackers compromised trusted npm maintainer accounts via phishing, leading to the publication of malicious versions of 18 popular npm packages, impacting websites and applications by enabling crypto asset theft and the modification of developer tokens. Cloudflare's Page Shield, a client-side security solution, was highlighted for its ability to detect and prevent such attacks using a machine learning-based approach that processes vast amounts of JavaScript scripts daily, identifying less than 0.3% as malicious. The system employs a message-passing graph convolutional network (MPGCN) model, which learns to distinguish between benign and malicious code based on structure and syntax, making it resilient to various attack techniques. Improvements in the model's precision, recall, and F1 score have been achieved through enhanced training datasets and evaluation criteria. Despite the absence of detected activity related to the npm attack among Cloudflare users, the Page Shield successfully detected and blocked the threat, showcasing its capability to protect against fast-moving supply chain attacks. Going forward, Cloudflare plans to enhance its ML signals with contextual data and consolidate its classifiers by phasing out the "Code Behaviour Analysis" in favor of the more effective MPGCN-based approach.

The blog post delves into the performance issues encountered with BPF LPM tries, a critical data structure used for tasks like IP and IP+Port matching in network routing. These tries, integral to Cloudflare's Magic Firewall, have shown bottlenecks when dealing with millions of entries, leading to significant lookup times and CPU lockups, which can result in traffic packet loss. The post provides a refresher on trie data structures, explaining their efficiency in memory usage and search operations due to their prefix matching capabilities. It highlights the limitations of the current BPF LPM trie implementation, such as its two-child node design, which impacts search efficiency and height, and the lack of optimizations like level compression. The article shares benchmark results illustrating the declining performance with increased entries due to cache and dTLB miss rates. It concludes by discussing the need for improvements in the BPF LPM trie implementation, suggesting the adoption of a Level Compressed trie similar to those used in the Linux kernel for more efficient IP routing, with future work aimed at enhancing the lookup function's performance.

Modern applications, often composed of multiple distributed components, require sophisticated health assessments to ensure overall availability and performance. Cloudflare has introduced Monitor Groups for its Load Balancing service, a feature that allows for the aggregation of multiple health monitors into a single entity, providing a comprehensive view of an application's health. This system enables users to define critical components, utilize observational probes, and apply quorum-based health assessments, ensuring that failover decisions are both intelligent and resilient. Health checks are conducted globally, leveraging Cloudflare's extensive network to prevent localized issues from causing unnecessary failovers. The Monitor Groups feature, initially available to Enterprise customers via API, is designed to enhance the accuracy of health checks without the need for custom aggregation services and will soon be accessible through the Cloudflare Dashboard for all users. This development is part of Cloudflare's ongoing efforts to simplify traffic management and enhance load balancing capabilities for complex applications.

The concept of Web Application Integrity, Consistency, and Transparency (WAICT) is being developed to enhance the security of web applications, particularly those involving cryptography, by addressing issues such as code distribution and integrity verification. Unlike smartphone apps that benefit from app store security, web applications face challenges in ensuring the authenticity and consistency of delivered code. WAICT, supported by the W3C and involving browser vendors and encrypted communication developers, aims to create a framework that allows web applications to have similar security guarantees without centralized control. Key components include subresource integrity (SRI) and integrity manifests, which help define a coherent application by verifying assets and enforcing integrity across whole websites. Transparency is achieved by logging application versions in a publicly accessible, append-only log, with mechanisms in place to detect and monitor changes, ensuring that malicious modifications are auditable. The approach involves using hash chains for per-site logs and a prefix tree for managing enrolled sites, with a focus on maintaining openness, user privacy, and non-centralization. The system also introduces concepts like code signing for provenance and cooldown periods to prevent immediate malicious actions, with various roles like transparency services, witnesses, and asset hosts working together to maintain the system's integrity. WAICT aims to integrate additional security features over time and is in early stages of standardization, with collaborations ongoing with browsers and organizations like the IETF.

On October 4, independent developer Theo Browne published benchmarks comparing server-side JavaScript execution speeds between Cloudflare Workers and Vercel, revealing Cloudflare Workers initially performed worse than Node.js on Vercel by up to 3.5 times. Both platforms use the V8 JavaScript engine, leading to initial surprise at the results, but further investigation uncovered issues related to infrastructure tuning, JavaScript library differences, and benchmarking methodology. Cloudflare addressed these by optimizing routing algorithms, adjusting V8 garbage collector settings, and collaborating on improvements with the OpenNext project, leading to significant performance gains. While some disparities remain, particularly with Next.js, Cloudflare is committed to ongoing improvements and has begun submitting patches to enhance performance for everyone, including contributions to V8 and Node.js. The exercise highlighted the intricacies of benchmarking and the importance of representative testing, prompting Cloudflare to encourage developers to share benchmarks that might indicate performance issues, ensuring continued enhancements across platforms.

Cloudforce One aims to fortify internet security by offering a new suite of incident response and security services called REACT, which addresses internal network breaches that traditional edge defenses cannot fully mitigate. Despite Cloudflare's success in blocking significant numbers of cyber threats, real-world incidents like ransomware, insider threats, and application security breaches revealed critical gaps in response efforts due to fragmented internal team handoffs and delays. REACT, comprising seasoned security professionals, provides comprehensive support, including proactive threat hunting, incident readiness assessments, and rapid incident response, utilizing Cloudflare's extensive threat intelligence network. These services are vendor-agnostic and cater to diverse environments, ensuring swift threat mitigation and collaboration with executive leadership for informed decision-making. By integrating REACT services into the Cloudflare dashboard, customers benefit from seamless access to expert support for both proactive and emergency situations, enabling a unified defense strategy that bridges the gap between defense and recovery.

Cloudflare's team encountered a rare bug in Go's arm64 compiler due to their vast scale of handling 84 million HTTP requests per second, leading to the discovery of a race condition in the generated code. Initially, sporadic panics were observed in their arm64 machines, prompting a deeper investigation after these incidents increased without a clear cause. The panics were linked to a Go Netlink library used in their systems, which involved unsafe memory access and async preemption. The root cause was identified as a Go runtime bug, where async preemption during stack pointer adjustments led to crashes due to invalid stack pointers during garbage collection and stack unwinding processes. After isolating the issue and replicating it in a controlled environment, the bug was reported and subsequently fixed in newer Go versions. This experience highlighted the complexity and challenge of debugging rare race conditions at such a large scale, emphasizing the importance of understanding low-level runtime mechanics in resolving such issues.