July 2022 Summaries

This blog post delves into the Linux networking stack, specifically focusing on memory and window management of the receiving side of a TCP connection. The author investigates how much data can be stored in the TCP receive buffer and how fast it can be filled. The exploration focuses on the receiving side of the TCP connection to understand how to tune it for optimal speed without wasting precious memory. The post discusses various factors affecting the receive buffer, such as metadata cost, window scaling, and coalescing mechanisms. It also highlights the importance of managing the memory budget effectively to avoid packet loss and latency spikes.

Cloudflare has deployed its data center in Hagatna, Guam, a strategic location that helps reduce latency for users in the region. The deployment is part of Cloudflare's mission to make the internet faster and more reliable by expanding its global network footprint. By adding new locations like Guam, Cloudflare brings its services closer to users, improving their online experience. The presence at Mariana Islands Internet Exchange (MARIIX) allows Cloudflare to peer with local participants, further enhancing connectivity in the region. This deployment not only benefits Guam's residents but also those in nearby archipelagos such as Federated States of Micronesia and Marshall Islands. The significant reduction in latency has led to improved user experience and increased internet usage in these areas.

Cloudflare has been using Kafka since 2014 and currently runs 14 distinct Kafka clusters across multiple data centers with roughly 330 nodes. The company uses Kafka to decouple microservices, enabling concurrent work on features and products. They have built internal tools to ease adoption of Kafka, including the Messagebus-Client library and a schema registry called Messagebus Schema. Additionally, they provide a connector framework for easy spinning up services that can read from a system of record and push it somewhere else. To ensure observability, automated metrics and alert creation are provided wherever possible. As usage of Kafka continues to grow at Cloudflare, the company remains committed to improving tooling around Messagebus and making it easier to use, customizable, and observe.

A crash in a development version of flowtrackd highlighted an issue with libxdp, specifically the AF_XDP part not being Linux network namespace aware. The blog post describes the debugging journey to find the bug and fix it. Flowtrackd is a volumetric denial of service defense mechanism that sits in the Magic Transit customer's data path and protects the network from complex randomized TCP floods. It uses the Linux kernel AF_XDP feature to transfer packets from a network device in kernel space to a memory buffer in user space without going through the network stack. The issue was resolved by retrieving the netns_cookie associated with the socket at its creation and adding it in the comparison operation. The fix has been submitted, merged, and backported in libbpf and updated in the Rust crate accordingly.

Cloudflare follows a docs-as-code approach to create and publish product documentation in Developer Docs. This involves using Git with a public GitHub repository for tracking changes and handling branching and merging, as well as relying on GitHub Actions and Cloudflare Pages for continuous integration and delivery (CI/CD). The company also accepts external contributions to its documentation through its cloudflare-docs GitHub repository. This approach has helped improve the quality of their documentation over time by leveraging feedback from both internal and external contributors, while ensuring that all changes adhere to their style guide and content strategy.

In June 2022, a massive HTTPS DDoS attack was mitigated by Cloudflare's systems, with the largest attack on record reaching 26 million requests per second. The attack was launched by a botnet called "Mantis," which consists of approximately 5,000 bots. Unlike traditional IoT-based botnets, Mantis uses hijacked virtual machines and powerful servers, making each bot more resourceful. Since its inception, Mantis has targeted almost 1,000 Cloudflare customers across various industries such as Internet & Telecommunications, News, Media & Publishing, Gaming, and Finance. The majority of the attacks have been directed towards US-based companies, followed by Russia-based ones. To protect against Mantis and other DDoS attacks, Cloudflare's automated DDoS protection system leverages dynamic fingerprinting to detect and mitigate such threats.

NASA, ESA, and CSA released the first images from the James Webb Space Telescope (JWST), which conducts infrared astronomy to reveal the unseen universe. The telescope's first deep field image depicts a galaxy cluster with a distance of 5.12 billion light-years from Earth. Internet traffic analysis revealed an increase in general internet traffic in the US on July 11 and July 12, compared to the previous week. DNS request trends showed two clear spikes: one around the time the first galaxy cluster infrared image was announced by Joe Biden on July 11, with traffic rising 13x higher than in the previous week; and another during Tuesday, July 12, when more images were revealed, with a peak at 10:00 EST, with traffic being 19x higher than in the previous week.

Cloudflare has successfully planted the first 25,000 trees as part of its commitment to help clean up bad bots and improve the environment. Working with One Tree Planted (OTP), these trees have been used in the restoration of 20 hectares of land at Victoria Park in Nova Scotia, Canada. The planting projects added various tree species to two areas of the park, one previously affected by insect attacks and another that was a former municipal dump site. This initiative is part of Cloudflare's broader sustainability efforts, which include investing in renewable energy and tools to help customers understand and mitigate their carbon footprints on its network.

Cloudflare has introduced a new feature called "Waiting Room Event Scheduling" for its enterprise customers, allowing them to schedule changes to their waiting room's settings and custom queueing page ahead of time. This feature is designed to help manage traffic during online events such as seasonal sales, open registration periods, product drops, ticket sales, etc. The new queuing methods - Reject and Passthrough - give customers more options for controlling user flow before, during, and after their online events. Event Scheduling also includes a pre-queue feature that adds extra protection against traffic spikes that can cause a site to crash.

Cloudflare introduces Location-Aware DDoS Protection in beta for its Enterprise customers subscribed to the Advanced DDoS service. This new feature leverages machine learning models to identify traffic patterns and locations, helping detect and mitigate unwanted traffic from unexpected sources. The system is designed to be customizable, allowing users to adjust sensitivity levels based on their specific needs. Cloudflare's mission is to help build a better internet by making the impact of DDoS attacks a thing of the past through intelligent protection tailored to individual needs.

In July 2022, João Sousa Botto discussed the benefits of Signed Exchanges (SXG), a web platform specification that allows deterministic verification and faster serving of cached website versions. Google's experiments have shown an average reduction in Largest Contentful Paint (LCP) by 300ms to 400ms with SXG-enabled prefetches, leading to improved SEO and lower bounce rates. Cloudflare's Automatic Signed Exchanges simplifies the setup process for users. Real-world data from 500 sites showed that 85% experienced LCP improvements, while 78% saw First Contentful Paint (FCP) enhancements and 98% had Time to First Byte (TTFB) improvements. Overall, customers witnessed a median improvement of over 20% in these metrics. SXG is now supported by Chromium-based desktop browsers, including Chrome, Edge, and Opera, potentially doubling its coverage for site visits.

The US National Institute of Standards and Technology (NIST) has announced the post-quantum cryptography it will standardize, with Kyber chosen for key agreement and three signature schemes - Dilithium, Falcon, and SPHINCS+. This is a significant step towards securing the internet against potential quantum computer threats in the future. The selected algorithms offer good performance characteristics and security levels, making them suitable for practical deployment on the internet. Cloudflare has been working on implementing post-quantum cryptography and contributing to standards, and they are excited about this development. They also provide guidance on how developers can start experimenting with these new algorithms in their software stacks.

On July 8, 2022, an outage occurred at Rogers Communications, one of Canada's largest ISPs, impacting businesses and consumers. The incident began around 08:45 UTC and lasted for over 17 hours. Cloudflare data revealed a significant spike in Border Gateway Protocol (BGP) updates after 08:15, reaching its peak at 08:45. This suggests that the outage was likely due to an internal error rather than a cyber attack. The incident resulted in a near complete loss of traffic from Rogers ASN, AS812. As of July 9, 2022, at around 01:00 UTC, internet traffic started to come back gradually.

Cloudflare has expanded its Web Application Firewall's threat intelligence capabilities by adding four new managed IP lists. These lists, which include "VPNs", "Botnets, Command and Control Servers", "Malware" and "Anonymizers", can be used as part of any custom firewall rule. The managed lists are created and maintained by Cloudflare based on threat intelligence feeds collected from patterns and trends observed across the internet. Enterprise customers can use these lists in their WAF rules to manage incoming traffic from these IPs, with updates made multiple times a day.

Cloudflare has announced experimental support for WASI (WebAssembly System Interface) on its Workers platform, along with wrangler2 to make it easier to work with. WASI aims to provide a standard interface that any language compiling to WebAssembly can target. This enables developers to write applications that not only interoperate with any WebAssembly runtime implementing the standard but also any POSIX compliant system. The announcement includes examples of how this works in practice, such as deploying CLI apps in the cloud and using WASI from within a JavaScript Worker.

In Q2 2022, the largest HTTPS DDoS attack ever recorded occurred with a peak of 26 million requests per second. Broadcast media companies in Ukraine were the most targeted by DDoS attacks during this period. Banking, Financial Services and Insurance (BFSI) companies in Russia experienced the highest number of application-layer DDoS attacks. Ransom DDoS attacks increased by 11% quarter over quarter. Network-layer DDoS attacks also rose significantly, with a 109% increase year over year. The Telecommunications industry was the most targeted by network-layer DDoS attacks.

Cloudflare has announced improvements to its Page Shield product, which protects website owners and visitors against malicious third-party JavaScript. The enhancements include a new status field for scripts, metadata in alerts, and categorization of threats. These updates aim to help users focus on the highest impact scripts and get more value out of the product. Additionally, Cloudflare is working on developing attack type-specific detection mechanisms and overarching tools to detect anomalies.

During the second quarter of 2022, Cloudflare observed various internet disruptions across multiple countries due to different causes such as fiber cable cuts, maintenance work, and government-guided shutdowns. These incidents impacted millions of internet users and caused significant traffic drops in affected regions. Some notable events include Comcast's outage in Florida, Telkom SA's fiber damage in South Africa, suspected fiber cut affecting CANTV subscribers in Venezuela, AAE-1 & SMW-5 submarine cable cuts impacting connectivity across multiple countries, and internet shutdowns in Sudan, Syria, Algeria, and Iraq. Additionally, infrastructure issues like the fire at Costa Sur generation plant in Puerto Rico and maintenance work by Swisscom also caused disruptions. In Iran, several internet outages were observed due to mass protests and ongoing war in Ukraine led to near-complete internet outages in Kherson city.

John Graham-Cumming shares recent technical blog posts covering various topics such as the end of Microsoft's Internet Explorer and the rise of Edge browser, live-patching security vulnerabilities in Linux kernel with eBPF, Hertzbleed explained, Early Hints update for a faster internet, Private Access Tokens eliminating CAPTCHAs on iPhones and Macs, and optimizing TCP for high WAN throughput while preserving low latency. These blog posts cater to different interests including network security, web development, and system administration.

Cloudflare is constantly working on improving its service by monitoring and optimizing various performance metrics such as HTTP request latency and throughput. The company has addressed several issues related to TCP receive window size, buffer allocation, and collapse processing in order to provide world-class low latency while simultaneously providing high throughput over high-latency connections. By implementing a kernel patch that disables the TCP collapse logic and adjusting sysctl values, Cloudflare has achieved significant improvements in network performance for its products and services.