November 2024 Summaries

The Cloudflare Logs incident on November 14, 2024, resulted in a significant loss of customer event logs due to misconfiguration and cascading failures within the system architecture. The incident highlighted the importance of subsystems protecting themselves from failures in other parts of the larger system to prevent cascades. A bug in the configuration system led to an initial mistake, which triggered a second latent bug in Logfwdr itself, causing a massive spike in customer logs being sent by the service. The subsequent overload on Buftee, a buffer management system, resulted in it becoming unresponsive and unable to handle the increased workload. The incident serves as a reminder that failures within systems at scale are inevitable and require proactive measures to prevent recurrences, including regular testing and configuration of fail-safes and backup systems.

The BCS East-West Interlink submarine cable and the C-Lion1 submarine cable, both connecting countries in Northern Europe to each other, were damaged recently, but surprisingly, there was little-to-no impact on Internet connectivity in the affected countries, Lithuania and Sweden, Finland, and Germany. The significant redundancy and resilience of Internet infrastructure in Europe played a crucial role in minimizing disruptions. Multiple terrestrial fiber connections between neighboring countries also contributed to this resilience. In contrast, cable cuts often cause significant disruptions to Internet connectivity elsewhere, particularly when they represent a concentrated point of vulnerability for an individual network provider, city/state, or country. Building redundant and resilient network architecture is a best practice to mitigate these disruptions.

The world of Distributed Denial of Service (DDoS) attacks is evolving, with attackers leveraging cloud-hosted virtual machines to launch massive and sophisticated attacks. Over the last decade, DDoS attack sizes have grown exponentially, with a significant increase in requests per second, packets per second, and bits per second. The largest attacks now reach terabits per second, with some attacks reaching 3.8 Tbps. This growth is attributed to the increasing use of cloud-hosted virtual machines, which enable attackers to launch large-scale attacks without the complexities involved in infecting and managing fleets of IoT devices. Cloudflare has mitigated over 14.5 million DDoS attacks since 2024, with automation playing a crucial role in detecting and blocking these attacks. The company's Connectivity Cloud uses machine learning algorithms, global traffic distribution, and layered defense to protect against DDoS attacks, providing tailored protection for individual customers.

We've introduced a WebSockets API that establishes a single, persistent connection to the AI Gateway, enabling continuous communication and reducing the overhead of repeated handshakes and TLS negotiations. This API allows developers to access all AI providers supported by AI Gateway via WebSocket, with the option to authenticate requests using Cloudflare API tokens or custom headers in browser WebSocket clients. The new API uses Durable Objects to handle all messages for the connection, storing and applying AI Gateway settings passed via headers on a per-request basis. We've also introduced authentication for the AI Gateway WebSockets API, requiring a valid token included in the cf-aig-authorization header. This allows developers to maintain a single TCP connection between their client or server application and the AI Gateway, while ensuring security and scalability. The new API is now in beta and open to everyone, with support for streaming requests and handling asynchronous events through an eventId field.

Cloudflare is introducing Account Owned Tokens, which allow organizations to improve access control for their services by creating tokens owned by accounts instead of individual users. This feature enables more flexibility in representing what access should be used for, removing the need for manual configurations and ensuring that access remains independent of user lifecycle events. The new feature can be found on the "API Tokens" tab under the "Manage Account" section of the Cloudflare dashboard. Additionally, Cloudflare is launching Zaraz Automated Actions, a new feature designed to streamline event tracking and tool integration when setting up third-party tools. With this feature, users can automate sending data to their third-party tools without manual configuration, reducing complexity and saving time. Zaraz Automated Actions offers flexibility in automating common actions like page views, e-commerce events, and custom tracking, allowing users to fine-tune which actions are automated based on their needs. The new feature is available for all existing tools, and users can toggle it on or off from the tool detail page in their Zaraz dashboard.

Cloudflare has been formally verifying the correctness of its internal DNS addressing behavior using a custom Lisp-like programming language and a formal verification tool called Topaz. This process ensures that every possible DNS query for a proxied domain is mathematically proven to be handled correctly by the nameserver, which can help improve the reliability of the internet. The Topaz system executes a list of policies in sequence until a match is found, returning an IP address to the resolver. Cloudflare uses Topaz to manage its global DNS addressing behavior and ensure that different systems within the company have contradictory views on which IP addresses should be returned. The formal verification process has been implemented using Rosette, a solver-enhanced domain-specific language written in Racket, and has been shown to be effective in detecting bugs and conflicts between programs. Topaz's verifier is now deployed to production and formally verifies all changes made to the authoritative DNS behavior specified in Topaz.

The National Institute of Standards and Technology (NIST) has advanced 14 post-quantum signature schemes to the second round of its "signatures on ramp" competition, with a focus on their feasibility for use in Transport Layer Security (TLS). The goal is to develop a widely accepted standard for post-quantum signatures that can be used to secure online communication. The current top performers include HAWK, SNOVA, and MAYO, which offer improvements over existing algorithms like ML-DSA and Falcon. However, even these new schemes have limitations, such as requiring additional bytes on the wire. To mitigate this, experts suggest exploring alternative designs that reduce the number of signatures used in TLS, such as using a Key Encryption Mechanism (KEM) instead of a signature for handshake authentication or redesigning the vast majority of visits to use fewer online signatures. The industry is expected to continue working together to develop post-quantum security solutions without compromising performance.

Cloudflare observed a significant increase in Distributed Denial of Service (DDoS) attacks on political and election-related sites, with over 6 billion malicious HTTP requests blocked between October 31 and November 1. However, these attacks resulted in no significant disruption due to planning and proactive defenses. Internet traffic in the US increased on Election Day, with a 15% rise compared to the previous week, peaking after the first polls closed. DNS traffic for news outlets, polling services, and election-related websites also saw large increases, with peaks of up to 756% and 325% respectively. Email phishing trends showed a notable increase in emails mentioning candidates Trump and Harris, but at lower rates of spam and malicious emails compared to previous periods. Despite the rise in online attacks, Cloudflare's efforts to secure election infrastructure were deemed successful by the Cybersecurity and Infrastructure Security Agency.