October 2024 Summaries

The Cloudflare platform has launched a new feature called Workers Builds, which integrates Continuous Integration and Delivery (CI/CD) workflow into its Workers platform. This allows users to build and deploy full-stack applications and simple static websites with minimal developer overhead. The system is built on top of the Workers platform using Durable Objects, Hyperdrive, Workers Logs, and Smart Placement. The design problem was how to pick up a commit from GitHub or GitLab and start a containerized job that can clone the repo, build the project, and deploy a Worker. The solution uses a Client Worker to connect to a PostgreSQL database, a Build Management Worker to run and monitor builds, and Durable Objects to manage individual builds. Alarms are used to check for healthy startup and terminate builds that run longer than 20 minutes. Smart Placement and location hints reduce latency costs by serving requests close to the database. Workers Logs provide fast and easy-to-use logs directly within the Cloudflare dashboard, while JavaScript-native RPC enables almost no boilerplate code. The system has been tested using Vitest and workerd for unit tests and cross-worker integration tests. It is expected to be released soon with build caching, which will speed up customer builds by avoiding redownloading dependencies from NPM or rebuilding projects from scratch.

** Baselime, a platform for observability tooling, has successfully migrated to the Cloudflare Developer Platform, achieving significant cost savings and improvements in performance and scalability. The migration involved transitioning from an event-driven architecture on AWS to a serverless, globally distributed system built on Cloudflare primitives, including Workers, Durable Objects, and Workers Analytics Engine. By leveraging these services, Baselime reduced its cloud costs by over 80% and simplified its systems and codebase, enabling the team to focus on delivering product improvements. The migration also enabled real-time event ingestion, faster responses to users, and improved reliability. With this transformation, Baselime is now better positioned to deliver better observability tooling to its users.

OVHcloud experienced a brief but significant outage on October 30, 2024, which affected their backbone infrastructure and caused traffic to drop significantly. The outage was attributed to an issue in a network configuration mistakenly pushed by one of OVHcloud's peering partners, resulting in a BGP route leak that overwhelmed the network or its peer's network with traffic. An investigation revealed that Worldstream, one of OVHcloud's peers, experienced a similar issue due to a configuration error on one of its core routers, causing routes to be inadvertently advertised to all peers and leading to congestion on some paths. The incident highlights the importance of setting max prefix-limits for peers to prevent BGP route leaks and adopting measures such as Autonomous System Provider Authorization and Resource Public Key Infrastructure to protect networks from similar incidents in the future.

Cloudflare's DNS provider is used by 14.5% of all websites as an authoritative DNS provider, with a responsibility to provide reliable and high-performance service. The company's DNS system is often compared to a phone book, with records that provide details about domains, including IP addresses associated with hostnames. Cloudflare's scale means it handles a massive volume of DNS data, making migration from one database cluster to another a complex task. A team led by the DNS team migrated 20+ internal services from direct database access to using an internal DNS Records gRPC API, which allowed for better control over database access and improved performance. The migration involved several phases, including initial copying, change capture, and applying changes in the destination database. After the migration, Cloudflare saw a steep drop in usage across the original database cluster, and an increase in requests per second to the DNS Records API, with improved CPU usage and reduced incident rates.

The third quarter of 2024 saw a significant number of Internet disruptions, with various underlying causes including government-directed shutdowns, power outages, cable cuts, hurricanes, and military action. In Bangladesh, a nationwide Internet shutdown was imposed by the government to prevent cheating on exams, resulting in a near-complete loss of connectivity. Similarly, in Iraq's Kurdistan region, two sets of exam-related Internet shutdowns were observed, impacting multiple networks. In Syria, an exam-related shutdown was followed by another disruption caused by maintenance at a technical hall, while in Palestine, Israeli airstrikes likely knocked down local network provider Solidere's infrastructure. A fire near a data center in Algeria disrupted connectivity on AS327931 (Djezzy), and in Pakistan, a wide-scale Internet disruption lasted approximately two hours due to a disputed cause. In the United States, Verizon experienced a mobile network outage affecting some customers, while in the United Kingdom, Vodafone reported problems accessing both mobile and landline Internet connections. The third quarter's Internet disruptions highlight the importance of reliable electrical infrastructure and the need for governments to address underlying issues.

With September’s announcement of Hyperdrive’s ability to send database traffic from Workers over Cloudflare Tunnels, Hyperdrive has been designed to make the centralized databases feel like they’re global while keeping connections to those databases hot. This is achieved by using a global network to get faster routes to your database, keep connection pools primed, and cache your most frequently run queries as close to users as possible. To simplify this process, Cloudflare offers an excellent option for private networks: Tunnels. Hyperdrive handles Postgres traffic using an entirely custom implementation of the Postgres message protocol, allowing it to send messages across WebSocket streams without being bound to transport layer choices of some ORM or library. The approach relies on Rust traits and uses a mainstay of Rust - traits - to implement the necessary traits on top of a WebSocket stream, enabling Hyperdrive to use an existing WebSocket library to upgrade its SslStream connection to a Cloudflare Tunnel, which can then be used anywhere that other transport streams would work without any changes needed to the rest of the codebase. The solution is designed to help all those who want to use Hyperdrive without directly exposing resources within their virtual private clouds on the public web. This latest development has been adopted by multiple teams within Cloudflare, who are happily operating it in production today and finding great success building new or refactored products on Hyperdrive over Tunnels.

AI Gateway is a platform that addresses the challenges of managing multiple models and providers in the rapidly evolving AI landscape. With its ability to store and analyze billions of logs indefinitely, AI Gateway empowers developers and organizations to optimize their AI inference requests and responses in real-time. The platform's initial architecture faced limitations in log storage capabilities, but through technical challenges and strategic decisions, it was extended to meet the demands of today while being scalable for the future. The platform now features a centralized platform to efficiently monitor usage, implement controls, and gather data for optimization, as well as a system that manages thousands of Durable Objects to track user activities and ensure fair usage across all users. Additionally, AI Gateway is working on improving its Universal Endpoint with automatic retry capabilities, dynamic decisions based on request content, and enhanced resilience and flexibility in request management.

Workflows`, Cloudflare's durable execution engine that allows developers to build reliable, repeatable multi-step applications that scale for themselves, is now in open beta. Developers with a free or paid Workers plan can create and deploy workflows right away, without waiting lists or sign-up forms. Workflows are designed to make durable execution easy, persisting state and automatically scaling infrastructure. They provide a powerful platform for building long-running, multi-step applications that process user data, transform unstructured data into structured formats, export metrics, and automatically retry failed steps. With a simple-to-use API and first-party support in Wrangler, developers can create, manage, and observe workflows using various interfaces, including the REST HTTP API, Wrangler, Worker bindings, and the Web UI in the dashboard. Workflows is priced similarly to Cloudflare Workers, with CPU-based pricing that only charges for active CPU time and requests, not duration, making it an affordable option for building long-running applications. The open beta program invites developers to try out Workflows and provide feedback, shaping the roadmap ahead.

Cloudflare Queues are event-driven services that decouple Workers from the main application, allowing for greater scalability, reliability, and flexibility. The service uses Durable Objects as a building block for coordination and storage in a distributed system. Initially designed with a single Durable Object per queue, the v1 architecture had limitations such as latency, throughput, and consumer concurrency. A new v2 architecture was developed to address these issues by horizontally scaling across multiple Durable Objects, reducing latency, increasing throughput, and improving consumer concurrency. The v2 architecture also introduced a Coordinator and "Control Plane" to manage the individual Durable Objects in the system. With its performance improvements, Cloudflare Queues is now generally available, enabling developers to build more scalable and reliable applications on the Cloudflare Developer Platform.

BastionZero has joined Cloudflare, and they have introduced "Access for Infrastructure" as a native integration into their SASE platform, Cloudflare One. This feature enables organizations to apply Zero Trust controls in front of their servers, databases, network devices, Kubernetes clusters, and more. The first available feature is short-lived SSH access, with support for other popular infrastructure access target types like Remote Desktop Protocol (RDP), Kubernetes, and databases planned over the coming months. This integration aims to provide a unified experience that enhances security outcomes by applying Zero Trust principles to sensitive corporate resources, including infrastructure.

The 19th edition of the Cloudflare DDoS Threat Report reveals that the number of DDoS attacks increased by 49% quarter-over-quarter and 55% year-over-year in Q3 2024. Nearly 6 million DDoS attacks were mitigated, with over 200 hyper-volumetric attacks exceeding rates of 3 terabits per second (Tbps) and 2 billion packets per second (Bpps). The Banking & Financial Services industry was the most targeted sector. China experienced the highest number of DDoS attacks, while Indonesia was the largest source of such attacks.

The text discusses the challenges of detecting Distributed Denial of Service (DDoS) attacks and presents an anomaly detection pipeline developed by Cloudflare to identify unmitigated or partially mitigated DDoS attacks. The initial approach, based on a naive volumetric model, is shown to be ineffective due to its reliance on stable traffic volume over time, which rarely holds true in practice. Time series forecasting methods are also considered but deemed impractical for various reasons. The solution proposed by Cloudflare involves using multiple dimensions to measure traffic and identifying correlations between these variables. Through careful analysis, a dozen such variables were discovered that follow a normal distribution, aren't correlated with volume, and deviate from the underlying normal distribution during "under attack" events. Principal Component Analysis (PCA) is used to convert these multidimensional data into a spherical shape, allowing for an anomaly score based on distance from the center of the cloud. The process is highly parallelizable and can be scaled horizontally as needed. Cloudflare currently re-trains models every day but may reduce this frequency in the future due to minimal intraday model drift. The company trains models for a large sample of representative customers, including those on the Free plan, to identify attacks for further study and tuning of existing DDoS systems for all customers.

Cloudflare uses Baseboard Management Controllers (BMCs) to manage its servers' operations globally. The BMC is a special processor that ensures smooth server operation independently of the Central Processing Unit (CPU). Cloudflare customizes and deploys OpenBMC, an open-source firmware stack designed for various systems including enterprise, telco, and cloud-scale data centers. This open-source nature provides greater flexibility and ownership over this critical server subsystem, allowing faster development of custom features/fixes. However, while developing Cloudflare's OpenBMC firmware, the team encountered several boot problems such as servers not booting, missing memory modules, and issues with thermal telemetry. To address these issues, they implemented host ACPI state on OpenBMC to track various power state changes of the host during the boot process. They also fixed the issues by controlling power mode states for interfering devices, disabling BMC from reading DIMM temperature sensors during specific phases, and setting non-functional thermal sensors in their configuration. These improvements have led to better observability of the boot progress and "last state" of Cloudflare's systems, enabling more reliable operations and ownership over the firmware that powers its servers.

Vectorize is a globally distributed vector database built on Cloudflare's developer platform that enables users to create full-stack, AI-powered applications using Cloudflare Workers. It makes querying embeddings faster, easier, and more affordable by leveraging Cloudflare's global network, Cache, Workers, R2, Queues, Durable Objects, and container platform. Vectorize is designed for great index throughput and optimized for low query latency to deliver excellent performance for demanding applications. It supports a maximum of 5 million vectors, which is a significant improvement over the previous limit of 200,000 vectors.

Hono is a fast and lightweight web framework that runs anywhere JavaScript does, built with Web Standards. It was initially developed to create applications for Cloudflare Workers but has since gained popularity among developers due to its ability to run on multiple platforms such as Deno, Bun, and Node.js. The framework uses a Trie tree-based router and supports middleware and helpers that can be added when needed. Hono's strong type system allows for RPC (Remote Procedure Call) features, while server-side JSX provides an improved developer experience. With the release of version 4, Hono now supports static site generation, client components, and file-based routing, allowing users to create full-stack applications with a user interface.

Cloudflare has introduced a new integration with Okta to share risk indicators in real-time and simplify how organizations can dynamically manage their security posture in response to changes across their environments. The integration enables organizations to share Cloudflare's real-time user risk scores with Okta, which can then automatically enforce policies based on that user's risk. This allows for quicker adaptation to evolving risks and less manual effort. The integration is built on the OpenID Shared Signals Framework Specification (SSF) to ensure compatibility with current and future security providers.

Cloudflare's network spans over 330 cities in more than 120 countries, serving millions of HTTP requests and DNS queries per second. The company is focusing on developing and deploying more performant and efficient servers to support this growth. At the heart of each server lies the processor (central processing unit or CPU), which remains the biggest lever for improving performance. Cloudflare's 12th Generation server with AMD EPYC 9684-X (Genoa-X) is 145% more performant and 63% more efficient than its previous generation server. The performance gains come from various factors, including the number of cores, core frequency, thermal design power (TDP), and L3 cache size. The company conducted sensitivity analysis on three variants of 4th generation AMD EPYC processors to understand the contributing factors. Genoa-X 9684X with significantly more L3 cache offered the best performance among the three candidates, followed by Genoa 9654 and Bergamo 9754. In production, the Gen 12 server equipped with Genoa-X 9684X outperformed other options while consuming only 1.5x more power per system than its predecessor. This resulted in a 63% increase in performance per watt and improved cost structure for Cloudflare's infrastructure growth.

The author is excited to join Cloudflare as Product and Engineering lead, aiming to contribute to the mission of building a better internet. They have always been passionate about transformational technologies like Software-as-a-Service, cloud computing, and AI. They believe that innovation is crucial for technology companies and are inspired by creating products that achieve mass adoption. The author chose Cloudflare due to its unique position at the intersection of key technological trends such as AI development, reducing tech debt, multi-cloud deployments, and high performance connectivity. Their strategy for organic growth involves a platform approach focused on understanding customer pain points and building scalable solutions. They aim to further develop and refine Cloudflare's product offerings while prioritizing customers' needs.

The text discusses how the Site Reliability Engineering (SRE) team at Cloudflare built a self-healing platform using Temporal, a durable execution platform. This platform allows automatic remediation of failures in various components such as servers and software services, reducing toil and improving reliability for users. The system includes a coordinator for authorization and scheduling workflows, task routing for efficient task execution, and flexible trigger mechanisms for detecting failure conditions. Deployed in production, the platform has successfully mitigated potential impact from server-specific errors and reduced operational toil. Future plans include leveraging more Temporal features and exploring strategies to eliminate toil further.

Cloudflare has introduced two new features, DEX Notifications and China Express integration, to enhance its Cloudflare One platform's performance and security capabilities. DEX Notifications enable proactive monitoring of device health and application performance issues, while the China Express integration provides secure access to China-hosted sites for Cloudflare Gateway customers. These updates aim to improve user experience and maintain consistent zero trust security policies across global networks.

Kivera, a cloud security company, has joined Cloudflare to extend its SASE portfolio and incorporate inline cloud app controls. This acquisition aims to provide preventative security controls for all cloud services used by Cloudflare One customers. The integration of Kivera's technology with Cloudflare One adds preventive, inline controls to enforce secure configurations for cloud resources, offering enhanced visibility and granular controls. These new capabilities enable organizations to adopt a proactive approach in mitigating risks, managing cloud security posture, and embracing streamlined DevOps processes when deploying cloud infrastructure.

Cloudflare uses multi-tenant Kubernetes clusters across its core data centers for control plane, analytics, and various engineering tools. Multi-tenancy allows sharing of resources among multiple users or applications, reducing operational toil, costs, and increasing computational efficiency by sharing hardware. All of Cloudflare's Kubernetes clusters are multi-tenant with various components enabled for a secure and resilient platform. To address the need for virtualization, Cloudflare chose KubeVirt as it enables users to run virtual machines within Kubernetes, integrating seamlessly with containerized workloads on the same platform. KubeVirt is successfully running hundreds of virtual machines across several clusters at Cloudflare. The company uses KubeVirt for various internal projects such as Kubernetes scalability testing, development environments, kernel and iPXE testing, and build pipelines. KubeVirt has proven to be a great tool for virtualization in their Kubernetes-first environment, allowing them to support more workloads with their multi-tenant model. They are exploring additional features KubeVirt offers to improve the experience for their users.

The latest Gen 12 server hardware from Cloudflare is designed for efficient and reliable thermal performance across hundreds of cities worldwide. The company's robust thermal design includes strategic component placement, optimized heatsink design, and high-performing fans to ensure optimal cooling capacity. This new generation of servers also supports up to three times more total system power and is 100% air cooled for increased reliability. Furthermore, hardware flexibility is ensured through a reliable strategy that allows for the addition of GPUs and other accelerators as needed.

The text discusses the importance of security.txt, a standardized way for organizations to communicate with security researchers about potential website vulnerabilities. It highlights how Cloudflare has integrated this feature into their dashboard, making it simple and free for all users to enhance their security measures. The tool is designed for any Cloudflare user who manages a website, from small business owners to large enterprises. It provides an easy way to create and manage the security.txt file in the Cloudflare account, ensuring effective handling of vulnerability reports. The integration also supports optional fields such as encryption keys and signatures within the security.txt file for enhanced trust with security researchers.

Cloudflare has concluded its litigation against patent trolls Sable IP and Sable Networks with a victory in the U.S. District Court for the Western District of Texas. The company launched Project Jengo, an effort to fight back against patent trolls by seeking prior art that can invalidate any patents held by them. Since its launch in 2017, over $125,000 has been given out to individuals through this initiative. Sable IP and Sable Networks have now agreed to pay Cloudflare $225,000, grant a royalty-free license to their entire patent portfolio, and dedicate their patents to the public, ensuring that they can never be used against another company again.

Since early September, Cloudflare has been combating a month-long campaign of hyper-volumetric L3/4 DDoS attacks, mitigating over one hundred such attacks throughout the month. The largest attack peaked at 3.8 Tbps, which is the largest ever disclosed publicly by any organization. Cloudflare's autonomous detection and mitigation systems have been able to handle these monstrous attacks without impacting performance for their customers. The scale and frequency of these attacks are unprecedented, with the ability to take down unprotected Internet properties or those protected by on-premise equipment or cloud providers that lack sufficient network capacity or global coverage.

On September 30, customers on Verizon's mobile network in multiple cities across the United States experienced a loss of connectivity. The outage impacted Internet traffic on Verizon's network, with a slight decline (-5%) in HTTP traffic visible around 11:00 ET (15:00 UTC). Request volume fell as much as 9% below expected levels at 13:45 ET (17:45 UTC). The problem was acknowledged by Verizon on social media, and engineers worked to identify and solve the issue. Service started to be restored around 17:00 ET (21:00 UTC), with full restoration reported at 19:18 ET (23:18 UTC). Voice services may have been more significantly impacted than data services during the outage, but no specific information has been provided by Verizon regarding the root cause of the network problems.