November 2022 Summaries

Cloudflare is raising its prices for the first time in 12 years starting January 15, 2023. New sign-ups will be charged $25 per month for the Pro Plan and $250 per month for the Business Plan. However, existing customers who signed up before January 15, 2023, will continue to pay the old monthly price until May 14, 2023. The company is also introducing an annual payment option with discounts equivalent to the previous monthly prices. This change aims to align Cloudflare's business model with its underlying costs and accelerate network expansion and innovation.

Cloudflare has expanded its Cloudflare One product suite to improve performance, security, and reliability for users and network resources in mainland China. The company aims to create the best Secure Access Service Edge (SASE) experience for organizations operating in China. Through strategic partnerships with local providers, Cloudflare offers enhanced connectivity, security, and performance for networks and users within the country. Key features include accelerating traffic from China networks to private or public resources outside of China, enforcing uniform security policies across remote user traffic, and extending SASE filtering to local China data centers (in future).

In this post, we discussed the importance of protecting cryptographic keys and secrets from leaking into unauthorized memory regions. We talked about some real examples where data leakage has impacted many users around the world, including Cloudflare. Finally, we learned how the Linux Kernel Retention Service can help us to protect our cryptographic keys and secrets. We also introduced a working patch for OpenSSH to use this cool feature of the Linux kernel, so you can easily try it yourself. There are still many Linux Kernel Key Retention Service features left untold, which might be a topic for another blog post. Stay tuned!

Cloudflare has rarely discussed its egress network design, which is critical for its operation. The company manages Cloudflare IP addresses used to retrieve data from the internet and optimizes them for efficient use of available IP space. It uses a combination of anycast on ingress and unicast on egress, with each server having its own unicast IP address. To work around traffic segmentation issues, it added an untrusted WARP IP address to each server. The company also tags its egress IP addresses with country codes and publishes them as a geofeed. It has developed a solution called "soft-unicast" that allows sharing of a /32 egress IP among many servers by splitting the IP across servers using port ranges, improving utilization of IP space and reducing costs.

On Thanksgiving Day 2022, US Internet traffic dropped by 13% during dinner time compared to the week before. This drop was observed between 21:00 and 01:00 UTC (translated to East Coast as 16:00-20:00 EST and West Coast as 13:00-17:00 PST). After dinner, traffic increased by 15% compared to the day before. On Black Friday morning, traffic was growing ~16% more than on Thanksgiving Day. Mobile devices were responsible for an average of 55% of Internet traffic in the US on Thanksgiving Day, making it the day with the highest mobile predominance in November. E-commerce DNS query trends showed a clear dip during Thanksgiving dinner, with 13% less traffic than the previous week at 17:00 EST. Japan also experienced an impact on its Internet traffic due to Labor Thanksgiving Day, with a peak of 60% higher traffic than in the previous week.

The text discusses the Border Gateway Protocol (BGP), a signaling protocol used by autonomous systems on the internet to exchange information about IP prefixes and routing decisions. It explains how traffic engineering is done using prepending, which artificially increases the AS-path length to influence other networks' choices. The text also highlights that BGP is colloquially referred to as a Business Policy Engine because it selects the best path from a business point of view rather than performance. Furthermore, it discusses how community-based local preference can be used for ingress traffic engineering when prepending isn't effective or available.

Today, Cloudflare introduced its route leak data and API, allowing anyone to access information about route leaks across the internet. The system collects data from public sources and Cloudflare's view of the internet through its massive global network. Route leak detection results are now available on Cloudflare Radar's ASN pages and via the API. Route leaks can cause significant disruptions to internet users, as seen in past notable incidents. Despite their criticality, route leaks are often detected only when users report noticeable effects. To address this issue, Cloudflare has developed a system that automatically detects route leak events and sends notifications to multiple channels for visibility. The company plans to release more features related to routing security in the future.

Cloudflare has been named one of the Top 100 Most Loved Workplaces in 2022 by Newsweek and Best Practice Institute (BPI). The company strives to create a safe and inclusive workplace where employees can bring their whole selves to work. Key factors contributing to this recognition include Cloudflare's mission to help build a better Internet, fostering a diverse and supportive team culture, offering comprehensive benefits and flexible work arrangements, and prioritizing employee growth and success. With over 3,100 global team members, Cloudflare continues to expand its talent pool worldwide.

Cloudflare has introduced Workers Analytics Engine, a new tool that enables developers and product teams to build time series analytics about anything, with high dimensionality, high cardinality, and effortless scaling. The Analytics Engine SQL API is used by the company itself to monitor its own API, find bugs, and prioritize new product features. It allows users to generate events from Workers with just a few lines of code and query these events using GraphQL or SQL API for useful insights about business or technology stack. Since its open beta release in September, new features have been added based on developer feedback. The Analytics Engine is designed to handle high-cardinality data and can be used with any data visualization tool like Grafana. It currently offers free 90-day retention for users.

Cloudflare has released an updated version of @cloudflare/workers-types with new features and improvements to its automatic generation scripts. The latest release includes better interoperability with standard types, compatibility with different versions of the types, improved integration with Wrangler, and enhanced documentation and changelogs. Additionally, developers can now use TypeScript with Workers more easily by setting up a tsconfig.json file. Cloudflare encourages users to upgrade to the latest version and provide feedback on the new features.

Developer Week 2022 concluded with a series of 31 posts on Cloudflare's vision, product announcements, technical deep dives, and customer stories. Key highlights include the open beta launch of Queues for building performant distributed applications, the introduction of R2 Super Slurper for easy migration from S3, and the availability of D1 global relational database in open alpha. Technical deep dives covered topics such as improving standards compliance in Workers API, indexing millions of HTTP requests using Durable Objects, and server-side rendering with Pages Functions. Customer stories showcased how companies like DevCycle, Neon.tech, Xata, and Twilio leverage Cloudflare's technology for their infrastructure needs.

Miniflare 3, a local Cloudflare Workers simulator, has been released with improved accuracy and compatibility due to its use of the recently open-sourced workerd runtime. This new version simplifies the implementation by removing over 50,000 lines of code from Miniflare 2 while retaining features like live reload and detailed logging. Additionally, an experimental feature allows developers to access real data from Cloudflare KV namespaces locally. The Wrangler CLI will be updated in the future to make local development the default experience, offering significant performance improvements. The Workers team plans to focus on improving automated testing and creating IDE extensions for easier development of workers.

Cloudflare has shared an update on its Workers products' performance against competitors like Fastly and AWS. The company claims to offer the fastest developer platform on the market, with tests showing that it is faster in more networks than its rivals. Cloudflare also reported being the fastest provider in every country, further delivering on its mission to be the fastest network globally. The company's developer platform is built on top of the world's fastest network, ensuring users see the best experience possible.

Cloudflare has introduced Workers Logpush, a new feature that extends the functionality of wrangler tail by allowing developers to send their Workers logs to an object storage destination or analytics platform of their choice. This tool is now available for everyone on the Workers Paid plan and includes metadata about requests, console.log() messages, and uncaught exceptions. It supports popular observability tools like Datadog, New Relic, and R2 for storage and ad hoc querying. The pricing for Workers Logpush is $0.05 per million requests, with an included usage of 10M requests each month.

Twilio Segment has developed an Edge SDK built on Cloudflare Workers to solve problems encountered with tracking first-party data and personalization. The SDK enables developers to collect high-quality first-party data and access real-time user profiles and state for delivering personalized app experiences without managing a ton of infrastructure. The Edge SDK is currently in early development, with plans to launch a private pilot and open-source it in the near future.

Cloudflare has announced Snippets, a new feature that allows users to perform traffic modifications beyond the capabilities of its existing productised offerings. Snippets will support the import of code written in various languages such as JavaScript, VCL and Apache .htaccess files, allowing customers to migrate legacy operational code onto Cloudflare's platform while consolidating their JavaScript operations. The majority of users will not pay extra for using Snippets, which are available at no additional cost across all plan levels. The feature is currently under development with a closed beta expected in early 2023.

Cloudflare has launched a new version of its website, Radar 2.0, which was built using the company's own products and services. The architecture of the site is divided into three main layers - Core layer, Cloudflare network layer, and Client layer. The new Radar features improved user experience, enhanced data insights, and better navigation. It also uses server-side functions for rendering and running the backend API. Remix, a popular framework for building web applications, was used in conjunction with Cloudflare Pages to optimize performance and reduce network traffic. Additionally, the site makes use of SVGs for images and graphical assets, which are scalable and efficient. The development process involved continuous integration and deployment, accessibility and localization considerations, and the use of Jupyter notebooks for data exploration and storytelling.

This post discusses a technique called "fragment piercing" which allows developers to replace selected parts of a legacy client-side rendered application with server-side rendered fragments. The result is an application where the most important views are interactive sooner, can be developed independently, and receive all the benefits of the micro-frontend approach, while avoiding large rewrites of the legacy codebase. This technique is framework-agnostic and has been demonstrated using React, Qwik, and SolidJS. The "Productivity Suite" demo application showcases how different independent technologies can be used for both the legacy application and for each of the fragments.

Cloudflare has announced that its Pages Functions feature is now generally available, allowing developers to easily add dynamic content to their static sites using the power of Cloudflare Workers. Since its open beta release in November 2021, Pages Functions has been fine-tuned based on developer feedback and usage patterns. The platform now offers improved features such as direct uploading of functions, enhanced debugging capabilities, real-time function metrics, integration with Cloudflare's ecosystem including storage products and secret bindings, compatibility with popular full stack frameworks like Next.js 13, and the ability to scale without limits.

In November 2022, Cloudflare announced a $1.25 billion funding program for startups building on its developer platform, Cloudflare Workers, in partnership with 26 leading VC firms. Today, the company revealed the first cohort of Launchpad Startups and added 14 additional VC partners, bringing the total potential funding to $2 billion from 40 VC firms. The Fall 2022 cohort of Launchpad startups features 25 diverse businesses that utilize Cloudflare's developer platform for building secure, reliable, and feature-rich products. Applications are open quarterly for startups interested in pitching VCs and joining the program.

Cloudflare has announced the general availability of Pages Functions, which enables dynamic server-side rendering for applications built using major meta-frameworks such as Astro, Next.js, Qwik, Remix, Solid, and Svelte. Built on top of Cloudflare Workers, Pages projects benefit from low latency with over 275 data centers across the globe. With server-side rendering, developers can optimize access to resources using Cloudflare's high-bandwidth connectivity and ability to cache data nearby, minimizing time spent on devices waiting for data. Many modern full-stack frameworks provide isomorphic code that works on both frontend and backend, avoiding duplication of business and data access logic.

Deployments for Workers have been introduced by Kabir Sikand on November 17, 2022. This feature allows developers to track changes made to their Worker, including code, configuration, and bindings. Deployments provide an audit log of changes to production applications, which can be used as a foundation for automated deployments, rollbacks, and integration with version control systems. The deployment information is accessible through the Cloudflare dashboard and Wrangler tool. Upcoming features include rollbacks, deploy status, deployment rules, view-only mode for historical deployments, and automation from commits to repositories via integrations with services like GitHub, Bitbucket, and Gitlab. Developers are encouraged to join the conversation on Cloudflare's Developer Discord.

Configuration management is crucial for organizations as they scale, and managing Cloudflare accounts can be challenging due to the numerous products and API endpoints. The use of Terraform can help manage complex Cloudflare configurations by providing a secure, auditable, and self-service way of updating system settings. By defining configuration in code using Terraform, developers and users can make changes without needing an administrator's approval. Using Atlantis as CI/CD for Terraform ensures that modifications to accounts are peer reviewed, each change is tied to a user and commit, and account configuration can be audited by anyone at the company. Storing tfstates in encrypted datastores and using daily Terraform applies helps prevent unintended config drift and rotate certificates when they approach expiration.

In May 2022, Cloudflare announced its quest to simplify databases by building them, maintaining them, and integrating them. The goal is to empower users with the tools to run a powerful, scalable database without any hassle. To achieve this, they first aimed at reimagining the database development experience for every type of user. Over the past couple of months, they have been working on creating just that while learning important lessons along the way. They introduced an "open alpha" for their D1 product during Developer Week to gather feedback from users before making it production-ready. The open alpha includes features such as using the D1 dashboard, Wrangler CLI, accessing D1 from within a Worker or Pages Function, and community-built tooling like d1-orm, workers-qb, d1-console, and Kysely adapter for D1. Some limitations still exist in testing, such as database location, concurrency limitations, and availability limitations. They request feedback from users to improve the product for their production apps.

D1 is an SQL database service that allows developers to store and retrieve data quickly in their applications. It can be used alongside other tools like NoSQL and graph databases. D1 provides indexes for fast data retrieval, joins for complex relationships between tables, and a declarative approach to data validation, creation, and querying. The recent release of D1 in open alpha has made it an instant value-add to applications without needing new tools or stepping out of the Cloudflare ecosystem. It enables developers to create stateful applications locally and deploy them easily. In this blog post, the author demonstrates how to use D1 to add comments to a static blog site by creating a new D1 database and building a simple JSON API for comment creation and retrieval. The separation of D1 from the app allows for easier deployment and abstraction of the static and dynamic pieces of the website.

Cloudflare has announced the general availability of OpenAPI Schemas for its API. These are published via GitHub and will be updated regularly as new APIs are added or existing ones are updated. OpenAPI is a widely adopted standard for defining APIs in a machine-readable format, allowing for integration with various tooling to accelerate development. The adoption of OpenAPI by Cloudflare provides benefits such as easier maintenance and updates of its APIs, and the ability to leverage numerous tools from the OpenAPI community.

The Workers Browser Rendering API is a new out-of-the-box solution for simplifying browser automation tasks such as capturing images or screenshots. It enables developers to programmatically perform actions that users can do when interacting with a browser, making it easier to create tools like taking automated screenshots and testing UI/UX flows. The API is currently in closed beta and interested developers can join the waitlist.

Cloudflare is set to introduce mutual TLS authentication support in its Workers product, allowing workers to send outbound requests through a mutually authenticated channel. This feature will provide an additional layer of security for applications built on Workers by enabling two-way authentication between the client and server. The implementation of mutual TLS will allow users to upload their own client certificates and attach them in fetch() requests within a worker, providing granularity at both the worker and request levels. This highly requested feature is expected to be launched soon, making it easier for developers to use Cloudflare as their platform for building applications.

Xata, a modern data platform designed for scalable applications, has built its serverless functions product, Xata Workers, on top of Cloudflare's Workers for Platforms. This enables developers to deploy functions that sit in front of their databases while benefiting from the security, scalability, and performance of Cloudflare Workers. Xata Workers are designed to simplify database access and allow developers to focus on application logic without worrying about maintaining underlying infrastructure. The platform is currently out of beta and offers a generous free tier for users to build and deploy applications, with additional features such as branching, search, analytics, transactions, and more.

Cloudflare has released an updated version of its API reference documentation, standardizing content and improving the developer experience when interacting with Cloudflare's API. The new API documentation is designed to meet users where they are, providing clear, easy-to-use products in the dashboard for most customers. However, developers often prefer using a CLI or Wrangler for more control over their Cloudflare products and deeper ties into their company's internal applications. The updated API docs include improved navigation, clarified authentication information, and examples to minimize developer friction when deploying code. The new site also supports mobile access and includes keyboard shortcuts for search functionality.

In May 2022, Cloudflare announced its quest to simplify databases by building, maintaining, and integrating them. The company aims to empower users with tools for running powerful, scalable databases without hassle. During the past couple of months, they have been working on reimagining the database development experience for all types of users, not just experts. The D1 open alpha has been launched as a new concept for developers. It is designed to help users get a feel for the product on their hobby apps or side-projects before it becomes fully production-ready. The alpha includes features such as using the D1 dashboard, working with the Wrangler CLI, accessing D1 from within Workers and Pages Functions, and utilizing community-built tooling like d1-orm, workers-qb, d1-console, and D1 adapter for Kysely. However, some crucial pieces are still in active development and testing, such as replication and JavaScript transaction support. Limitations include database location, concurrency limitations, and availability limitations. Feedback is requested from users to improve the product for their production apps. Developers can join the #d1-open-alpha channel in the Cloudflare Developers Discord or create threads in the Cloudflare Community forum under the D1 category.

Cloudflare has introduced an open beta for its new feature, Cache Reserve, which helps users serve content from its cache for longer by using R2's persistent data storage. This feature benefits website operators by reducing their bills for egress fees from origins and benefiting website visitors with faster content loading times. Cache Reserve is designed to be used with Tiered Cache enabled for maximum origin shielding, and it accomplishes these origin egress savings for a low price based on R2 costs. The company plans to add more features and integrations across Cloudflare in the future.

The text introduces R2, an S3-compatible globally distributed object storage that allows developers to store large amounts of unstructured data without incurring costly egress bandwidth fees. It announces the launch of the R2 Super Slurper feature, which enables users to move all their data to R2 in one giant slurp or sip by sip through a user-friendly interface and API. The first iteration allows for a simple, one-time import from an S3 bucket into an R2 bucket. Future developments include automatic sip by sip migration, allowing users to start serving objects from R2 as they migrate, saving money immediately. Currently, the feature targets S3-compatible buckets, with other sources expected to become available in 2023. To access the private beta of the R2 Super Slurper, users must be R2 users and sign up for the waitlist.

Cloudflare has introduced a collection of ready-made templates for its developer platform, Workers. These templates are designed to help developers get started quickly and focus on building and innovating their applications without spending much time configuring and setting up projects. The templates showcase different use cases of Workers and highlight the possibilities of building applications using Cloudflare's developer ecosystem. Some examples include an image sharing website, a direct creator upload to Cloudflare Stream, and a Durable Object-powered request scheduler. These templates also come with additional features such as integrated Deploy with Workers button, support for Test Driven Development, and online IDE integration with StackBlitz. Developers can contribute their own templates to the collection by sending in a pull request.

Cloudflare has introduced a new feature that allows users to store and retrieve logs using R2, which is more cost-effective than traditional methods. The company built an API for log retrieval using Cloudflare Workers with Durable Objects and the Streams API. This approach enables customers to index and query millions of their Cloudflare logs stored in batches on R2 without needing to manage their own database or logging infrastructure. The system uses Durable Objects to maintain indexes of data stored in R2, simplifying the process for developers. Currently, RayID lookups are supported for HTTP Requests, Firewall Events, and soon Workers Trace Events. Cloudflare plans to add more complex types of filters and queries that users can run against their logs in the future.

Postgres is one of the world's favorite databases due to its ease of use and scalability. However, connecting it from environments like Cloudflare Workers presents challenges. Neon.tech not only solves these problems but also offers other features such as branching databases. To get started with Neon, users can install the client library @neondatabase/serverless, which is a drop-in replacement for node-postgres. The solution consists of connection pooling built into the platform, a WebSocket proxy, and a client library that supports end-to-end TLS encryption between Workers and the database. Neon's serverless driver and WebSocket proxy are currently in public beta, with plans to improve and extend them further.

Cloudflare has introduced Logs Engine, a new product designed to help customers store, search, and understand their logs within the platform. The tool is aimed at enabling developers to debug applications, identify security vulnerabilities, or understand user interactions with products that use Cloudflare services. With Logs Engine, users can now store logs for any Cloudflare dataset for as long as desired and access them without relying on third-party tools. The company plans to add more powerful querying capabilities in the future, including filtering on any column and cross-dataset queries.

The text discusses the concept of "Supercloud," which is a new model for cloud computing that aims to make developers highly productive at scaling from one to Internet-scale. Unlike traditional cloud services, Supercloud focuses on granularity and efficiency by allowing code and data to move around the network, enabling automatic scaling and efficient resource allocation. The author argues that this approach makes it more economical and environmentally friendly while providing better performance for end users. Cloudflare has been building the Supercloud into their network and developer platform for years, with over a million developers currently using it.

DevCycle, a feature management tool for engineering teams, has experienced an interesting challenge while scaling its new product. The company's Cloudflare Workers-based infrastructure can handle more instantaneous load than their traditional AWS infrastructure, leading them to rethink how they design their infrastructure and use Cloudflare Workers for everything. DevCycle is built upon the experience using Workers which have served over 140 billion requests for Taplytics customers. The architecture of DevCycle revolves around publishing and consuming JSON configuration files per project environment, with high-scale requests managed by Cloudflare and keeping AWS services simple and low-scale. One of the primary challenges faced is handling instantaneous spikes in traffic to their APIs, which led them to move towards a new events Cloudflare Worker that can handle the load and make use of Kinesis Data Firehose for writing events to an existing S3 bucket. The company also advises relying primarily on data sent as part of the API request and cacheable data on Cloudflare's global network when architecting systems using Workers. The predictable cost structure of Workers has resulted in a more cost-efficient operation for DevCycle, with infrastructure spending directly tied to using their APIs and SDKs.

Cloudflare has launched an open beta of its new service, Cloudflare Queues, which aims to bring message queues to every developer building applications for Earth's regions. This innovative service follows other offerings like Cloudflare Workers and R2, enabling developers to build more complex applications without worrying about network configuration, region selection or capacity estimation. Unlike many other services, there are no significant egress charges associated with using Cloudflare Queues. Message queues are a fundamental building block of cloud applications, allowing for decoupling of application components and better management of batching and buffering calls to downstream services and APIs. They also enable easier reasoning about deployments, implementation, and feature shipping without worrying about synchronizing complex deployments. To get started with Cloudflare Queues, developers need to enroll in the open beta by navigating to the Workers section of their Cloudflare dashboard, selecting Queues from the navigation menu, enabling the Queues Beta, reviewing and completing the purchase details. Once enrolled, users can create a queue, manage it using Wrangler (CLI for Cloudflare Workers), and build applications that integrate Workers, Queues, and R2. The open beta roadmap includes ambitious goals such as supporting unlimited throughput while maintaining 100% durability, first-in first-out (FIFO) message processing, and API compatibility layers to ease migrations. Feedback from users will help shape the Queues roadmap as they continue to develop application integration services for the next generation cloud.

The WinterCG (W3C Community Group) has been working on identifying common API standards for all JavaScript runtimes to build upon. They have compiled a list of the "Minimum Common Web Platform API" which includes AbortController, ReadableStreamDefaultController, ByteLengthQueuingStrategy, SubtleCrypto, CompressionStream, TextDecoder, CountQueuingStrategy, and more. Workers runtime has made significant improvements in aligning with these standards by enhancing its implementations of Event, EventTarget, URL, streams, AbortController, AbortSignal, TextEncoderStream, TextDecoderStream, CompressionStream, DecompressionStream, and URLPattern APIs. The team plans to continue working on the implementation of fetch() and WebSockets APIs while seeking closer alignment with other runtimes through collaboration in the Web-interoperable Runtimes Community Group.

In the weeks leading up to the US elections, there was a significant increase in traffic to websites of state and local governments protected by Cloudflare's Athenian Project, as well as political campaigns and state parties using Cloudflare for Campaigns. SQL injection (SQLi) and HTTP anomaly attacks were the most common types mitigated by Cloudflare's Web Application Firewall (WAF). The majority of these attacks originated from IP addresses in the United States, although it is possible that attackers are using compromised systems within the country to launch their attacks. Despite the increase in overall traffic and attacks, WAF-mitigated traffic remained consistent throughout October and into November.

On November 8, 2022, the US held its midterm elections, with races for all 435 seats in the House of Representatives, 35 Senate seats, and gubernatorial races in several states. Cloudflare played a role in safeguarding the electoral process through their Impact programs, which provide cyber security products to protect access to authoritative voting information and sensitive voter data. The company has worked with state and local governments, political campaigns, and election results sites to ensure free and fair elections. During the midterm elections, Cloudflare did not identify large-scale attacks on November 8, 2022, indicating that robust cyber security preparations by various groups were successful.

Brazil's recent presidential election generated significant attention and impacted Internet traffic patterns in the country. On election days, internet traffic went down during the day as people went outside to vote. The runoff election on October 30 showed a bigger drop in traffic than the first round on October 2. Mobile device usage was also higher on weekends and election days. Election-related sites saw increased interest during both rounds of elections, with peaks at 16x and 17.4x more DNS queries compared to September baseline. Traffic for candidate websites showed a clear spike for Lula's domains when he was announced as the winner. News websites experienced higher traffic during the first round than the runoff, with peaks occurring earlier in the evening due to faster release of official results. Overall, real-world events like elections can have a significant impact on internet traffic patterns and human behavior online.

On November 1, 2022, OpenSSL released version 3.0.7 to address two high-risk vulnerabilities (CVE-2022-3602 and CVE-2022-3786) in the OpenSSL 3.0.x cryptographic library. These memory corruption issues could potentially allow attackers to execute arbitrary code on a victim's machine. However, Cloudflare is not affected by these vulnerabilities as it uses BoringSSL in its products. The vulnerabilities reside in the X.509 certificate verification code and require specific conditions to be met for exploitation. Users should patch vulnerable OpenSSL packages and update their software to mitigate potential risks.