March 2020 Summaries

On April 1, 2018, Cloudflare launched its public DNS resolver, 1.1.1.1, with a focus on privacy and security. The company committed not to retain any personal data about requests made using the resolver and implemented technical measures for DNS over HTTPS to keep queries secure. To prove their commitment to privacy, Cloudflare engaged an independent accounting firm to examine its 1.1.1.1 resolver privacy commitments. The examination process provided valuable lessons on privacy by design principles and the importance of clear communication about data handling practices. As a result, Cloudflare refined its commitments to ensure that it does not track individual users' online activities or retain their source IP addresses in non-volatile storage. The company encourages other providers to adopt similar data handling practices for improved privacy and transparency for Internet users globally.

Cloudflare's network processes over fourteen million HTTP requests per second globally. To manage these high volumes of traffic, the company relies heavily on its configuration distribution system that needs to be fast and reliable. Historically, this system was built on top of Kyoto Tycoon (KT) datastore but as the company grew, KT started showing its limitations. The team decided to build a new replicated key-value store called Quicksilver tailored for their needs. Quicksilver uses LMDB as its storage engine and implements a fan-out type distribution mechanism for changes in customer configurations. It also includes features like transaction logs, batch mode updates, and optimizations such as compression and incremental hashing to improve performance and reduce disk space usage. The system is monitored using Prometheus and Grafana, with alerts distributed via PagerDuty. Cloudflare plans to open-source Quicksilver in the near future.

Cloudflare has transitioned from using traditional VPNs to a zero-trust architecture for remote work. They have migrated services one by one, focusing on high priority ones first and moving them to Access, their zero-trust access proxy. Some services that did not run over HTTP or other Access-supported protocols required the use of VPN, but support for SSH over Access has allowed them to replace the VPN as a protection layer for source control systems. They have also used Spectrum, their DDoS protection and performance product, to protect their VPN endpoints against DDoS and improve performance for VPN users. As of 2020, new employees no longer get a VPN account by default, indicating the company's progress towards completely adopting zero-trust architecture.

Cloudflare has launched a program offering its Cloudflare for Teams suite for free to any company until September 1, in response to the increasing need for remote workforces. The company's next-generation VPN alternative, Cloudflare Access, was developed as an internal solution to improve security and efficiency. Originally known as EdgeAuth, it began with one application protected by Access, but its adoption grew organically within the organization due to its effectiveness in streamlining authentication flows for popular internal tools. The zero-trust model of Access allows for granular permissions and increased network visibility, making it a valuable tool for securing remote work environments.

Cloudflare maintains PCI DSS compliance as a Level 1 Merchant to ensure secure transmission and processing of cardholder data for its customers. The company's products, including Web Application Firewall (WAF), Content Delivery Network (CDN), Cloudflare Access, and Time Service, are certified under the latest Attestation of Compliance. These services help customers meet their own compliance obligations and protect web-facing applications, secure internal resources access, provide accurate time services for audit logging, and enable higher levels of TLS encryption.

Victoria Bernard, an engineer at Cloudflare, shares her experience migrating developer documentation from Hugo to Gatsby. She explains why she chose Gatsby over Hugo and how the process unfolded. The main reasons for switching were the ability to handle more complex logic with React and the desire to improve the site's performance and SEO score. Bernard also provides a step-by-step guide on how to port markdown files from Hugo to Gatsby, focusing on creating pages and handling sidebar components. She concludes by sharing some lessons learned during the migration process and expressing her satisfaction with the results.

In this blog post, we discuss how we optimized disk encryption performance in our data centers using a combination of kernel patches and custom configurations. We achieved significant improvements by eliminating unnecessary queuing and asynchronous behavior in the dm-crypt module and synchronizing Linux Crypto API calls. The resulting changes effectively doubled the throughput and halved the latency for encrypted disk operations, while maintaining strong security guarantees. The performance improvement is particularly important for our caching infrastructure, where worst-case response times have been significantly reduced without any impact on overall cache performance. We believe that these optimizations can benefit other users of Linux disk encryption as well, and we are working towards including them in the mainline kernel source tree.

The Bandwidth Alliance, founded in 2018 by several cloud and networking partners, aims to help mutual customers reduce or eliminate data transfer charges between the cloud and consumers. With the addition of Alibaba, Zenlayer, and Cherry Servers, the alliance now has a total of 20 partners offering diverse cloud services and products. The alliance helps businesses avoid being locked into a single provider due to data charges from moving workloads between vendors. Nodecraft and Pippa are examples of companies that have significantly reduced their egress costs by joining the Bandwidth Alliance. A calculator is available for potential customers to estimate their savings.

Cloudflare developed a memory-SSD hybrid storage system that reduces cache hit tail latency and extends the lifespan of SSDs. The system puts unpopular assets in memory to reduce wear on flash memory chips, which are costly to replace. By not caching one-hit-wonders (assets accessed only once), disk writes were reduced by half and corresponding disk hit tail latency was reduced by approximately five percent. A transient cache in memory was implemented, where every asset is missed only once before being promoted to the permanent cache backed by SSDs if popular enough. This approach has been deployed in production, resulting in measurable decreases in cache hit tail latency and a 25% reduction in disk writes at peak usage. Future work includes broader deployment, smarter promotion algorithms, and exploring hybrid storage systems across memory, SSDs, and HDDs.

Cloudflare's security team has adopted a "builder first mindset" for new projects, using their own products and services wherever possible. They have launched support for the security.txt initiative through their serverless platform, Workers, to provide a common location for security researchers to learn about communication with their team. The Workers platform allows applications to be deployed on Cloudflare's edge network, providing high performance and reliability without requiring maintenance of servers or software stacks. This has enabled the development of various security projects, such as secure code review, CSP nonces and HTML rewriting, and authentication for legacy applications. The team also implemented support for security.txt on their website using Workers, automating deployment and updates while ensuring version control and easy maintenance. They have open-sourced the Worker itself to help others achieve similar security wins with the platform.

Cloudflare has introduced a new feature, the fallback pool, for its Load Balancer tool. This feature ensures that traffic is still routed even if all pools are down by directing it to a "pool of last resort." The company has also made updates to its dashboard UI, including adding the fallback pool option and improving other areas of the Load Balancing app. These changes include a facelift for the main Load Balancing page, de-modaling content from modal screens, and adding more information in tables to clarify shared objects among teams. Additionally, Cloudflare has added confirmation modals and updated CTAs/buttons to prevent unwanted changes and improve user confidence when making updates.

On March 20th at 4:00 PM, Connor Peshek introduced the thirteenth edition of "The Serverlist". This publication provides updates on serverless technology, developer tutorials, discussions with other serverless developers, and information about upcoming meetups and conferences. To receive this content directly in your email, sign up using the provided link while ensuring that your privacy is maintained.

Cloudflare has developed an internal video streaming platform called CloudflareTV, which is designed for secure access by employees only. The platform aims to organize and categorize all video content produced by the company, making it easily accessible from a browser. To ensure security, videos are locked down with signed URLs that require temporary tokens for access. Additionally, videos can be tagged for easy organization and searching. CloudflareTV is built using Workers and Stream, allowing for efficient encoding, storage, and delivery of video content without the need for separate infrastructure. The platform also integrates with Cloudflare Access to restrict access to authorized employees only.

This article discusses an unexpected solution to sporadic latency spikes experienced by Cloudflare's edge stack. The author explains that disabling keepalives between SSL and FL (two parts of their edge stack) improved the 99.9th latency percentile by 4x. Although generally beneficial, in this specific case, the local connection establishment was faster than the delay caused by landing on a busy worker. This solution allowed Cloudflare to run their machines hotter without seeing increased latency, reducing their overall footprint.

The author discusses their experience using Cloudflare Gateway to block distracting websites during work hours while working from home. They provide a step-by-step guide on setting up the service, including creating a policy for blocking specific sites and testing its effectiveness. The author also mentions that category-based DNS filtering can be used as an alternative method of blocking domains associated with certain categories. Overall, Cloudflare Gateway is praised for its ease of use and effectiveness in reducing distractions during work hours.

The article discusses the event loop architecture used by NGINX and its advantages and limitations for handling HTTP requests. It explains that an event loop works well when every piece of work finishes quickly but can be problematic if a task takes too long, causing other requests to be blocked. The author highlights issues with Cloudflare's workload, particularly the Web Application Firewall (WAF), which requires more CPU and could slow down other unrelated requests. The article explores various solutions to mitigate this issue, such as increasing worker processes, creating a separate service for WAF, or offloading CPU-intensive tasks to a thread pool. The author explains that NGINX already uses thread pools for filesystem operations and repurposed this system to offload the WAF processing. The article presents performance metrics before and after offloading the WAF into thread pools, showing significant improvements in Time To First Byte (TTFB) and accept latency. The author concludes that event-loop-based processing is advantageous in many situations but can be vulnerable when requiring a lot of CPU time. In such cases, using a separate service or offloading tasks to a thread pool may be the best tradeoff.

The COVID-19 pandemic has led to a significant increase in internet traffic as people rely on it to stay connected with friends and family, work from home, and access news and entertainment. Video conferencing, streaming services, and e-commerce websites have seen surges in usage. Internet traffic patterns are changing, with residential broadband networks seeing growth while business and university networks experience a slowdown. Entire countries are watching their leaders' announcements on the pandemic, leading to spikes or dips in internet traffic. Some national research networks have become almost as quiet as weekends due to university closures. Internet Exchange Points have also seen increases in traffic. The core of the Internet remains robust despite occasional service outages. Traffic is shifting from corporate and university networks to residential broadband, but the Internet was designed for change.

In December 2019, Cloudflare extended its analytics platform to include Business and Pro customers, providing them with powerful tools previously only available for Enterprise users. The analytics platform has been expanded to cover additional services such as Account Analytics, DNS Analytics, Load Balancing Analytics, and Monitoring Analytics. In January 2020, the company released Network Analytics dashboard for Magic Transit and Bring Your Own IP (BYOIP) customers, offering near real-time visibility into network- and transport-layer traffic patterns and DDoS attacks blocked at Cloudflare's edge in over 200 cities worldwide. The analytics data is stored efficiently to provide insights for up to one year back. Network Analytics are also available via GraphQL API, allowing customers to build custom reports and dashboards. Through interviews with internal customer-facing teams and external network and security engineers, two main use cases were identified: Incident Response and Reporting. The dashboard provides metrics that matter most for incident response, such as top destination IPs and ports, source IPs, ASNs, data-center locations, and protocol distribution. It also enables customizable DDoS attack logs and packet/bit distribution by source & destination, TCP flag distribution, and real-time packet and bit rates. For reporting purposes, the dashboard offers a one-line summary reflecting recent activity, dynamic maps showing geographic distribution of packets and bits based on Cloudflare data-center locations, and customizable filters for time range, IPs, ports, and more.

Cloudflare is committed to providing transparency and awareness about its response to the COVID-19 pandemic. The company has a Business Continuity Team that monitors the situation closely, ensuring minimal disruption to services. Offices in the US, EMEA, Sydney, and Singapore are physically closed with employees working remotely. In Beijing, employees alternate between working from home and the office. Cloudflare does not anticipate any significant impact on its products or services due to the pandemic. The company has taken steps to work with critical business partners and suppliers to ensure minimal disruption to customers. Anycast network ensures traffic failover in case of data center downtime, and Cloudflare's Customer Support Team is fully operational for communication during service interruptions.

The article discusses the changes in Internet usage patterns due to the COVID-19 pandemic. It presents data from Seattle, Northern Italy, and South Korea showing increased overall internet use, more extended hours of activity, and shifts in types of online activities such as increased chat system usage, video streaming, news access, and gaming. The article also notes that despite these changes, Cloudflare's network is well-equipped to handle the increased traffic without any impact on its performance or security.

Cloudflare has introduced new secure DNS filtering capabilities in its Cloudflare Gateway product, which aims to protect teams from security threats without compromising performance. The solution addresses key challenges faced by businesses in managing and securing global networks due to architecture limitations and the inability to scale. It leverages Cloudflare's extensive network of data centers around the world to ensure fast and secure connections for users. Gateway's secure DNS filtering capabilities are built on top of 1.1.1.1, the fastest public DNS resolver globally. The product is available in both free and paid tiers, with additional features offered in the paid version.

Cloudflare's CEO Matthew Prince has assured customers that the company is fully operational during the coronavirus emergency, with its network operations center (NOC), security operations center (SOC), and customer support teams working remotely. The company is monitoring for changes in cyberthreats and Internet usage patterns, which have increased by up to 30% in some regions due to more people working from home. Cloudflare has extended its offer of free access to its Cloudflare for Teams product to help businesses support secure remote work during the crisis.

Cloudflare has open-sourced a plugin that enhances the experience of using its Cloudflare Access service with Sentry, a popular software engineering tool used for error tracking and management. The new plugin eliminates the need for users to log in twice when accessing Sentry through Cloudflare Access. It leverages JSON Web Tokens (JWTs) to confirm user identity before granting or denying access to sensitive resources. This seamless integration simplifies the process of connecting teams to Sentry, reduces the number of credentials needed to be managed by administrators, and provides a more efficient experience for end users.

Replicated, a 5-year old infrastructure software company, has developed a solution to simplify the management of local development environments by moving them to Infrastructure as a Service (IaaS) resources using Cloudflare Access and Argo. The new cloud-based environment reduces troubleshooting time for developers and allows every engineer on the team to maintain a full stack development environment, even without deep Kubernetes expertise. This approach has resulted in better developer productivity and reduced dependence on powerful laptops. Replicated's experience with this solution demonstrates how moving development environments to the cloud can improve efficiency and security for software development teams.

Cloudflare is offering its Cloudflare for Teams products free to small businesses worldwide for at least six months to help ensure business continuity during the COVID-19 pandemic. The initiative aims to support remote workers by providing secure and easy access to online resources. Additionally, Cloudflare has launched an online hub where small businesses can find technology services available for free or a substantial discount from multiple companies during the Coronavirus Emergency.

The article discusses how traditional assumptions about work have changed, with employees now working remotely and using a variety of tools to do their jobs. This has led to challenges in productivity for remote teams, particularly when it comes to accessing internal applications and onboarding new contractors or part-time employees. To address these issues, the author suggests three steps: 1) replacing VPNs with Cloudflare's network to speed up remote teamwork; 2) using app launchpads to make tools easier to find; and 3) fast-tracking contractors by integrating multiple identity providers. The article also mentions that, until September 1, 2020, these features are being offered for free to small businesses implementing work from home policies due to the COVID-19 pandemic.

On International Women's Day and the Centennial Anniversary of the Nineteenth Amendment granting women the right to vote in the United States, Jocelyn Woolbright from Cloudflare shares five reflections. She highlights the long battle for women's suffrage, the delay in voting rights for women of color, the increase in women in political power globally, the evolution of sharing election information, and the importance of access to election information for voter turnout and democracy. Woolbright emphasizes Cloudflare's commitment to election security through projects like Athenian Project and Cloudflare for Campaigns, protecting state and local government election websites and providing services to U.S. Presidential campaigns.

In this article, Susan Chiang and Joe Sullivan share their experience of building a diverse security team at Cloudflare. They discuss lessons learned on how to attract and hire diverse talent, including choosing inclusive language for job postings, being open to multiple locations, leveraging various hiring sources, proactively reaching out to potential candidates, and utilizing internships to broaden the candidate pool. Additionally, they emphasize the importance of creating an inclusive culture by fostering psychological safety, promoting innovation, encouraging diverse perspectives, and celebrating successes. The article highlights how building a diverse team with a positive culture has helped their security team accomplish challenging tasks effectively.

Cloudflare, a company with over 1,200 employees in 13 offices and a network operating in 200 cities, has developed an alternative to traditional corporate VPNs. The company built Cloudflare Access as part of its Cloudflare for Teams security platform to replace its previous VPN system, which was difficult to manage and scale. Cloudflare Access runs on the company's serverless platform, Workers, allowing it to operate at a large scale without being constrained by a single physical appliance. The platform has been expanded over the last two years to address various use cases and remove reliance on VPNs. The Cloudflare for Teams security platform focuses on keeping users, devices, and data safe while maintaining performance and user experience. It replaces corporate VPNs with Cloudflare's network, allowing teams to deploy internal tools in any environment and secure them consistently using Cloudflare's network. Administrators can build rules to decide who should be able to access the protected tools, and users are prompted to authenticate with their team's identity provider when they need to connect. Argo Tunnel is used to connect resources through a secure outbound connection without exposing new holes in corporate firewalls. This tunnel makes outbound-only calls to the Cloudflare network, and organizations can disable all inbound connections by replacing complex firewall rules with just one rule. Cloudflare Access has been implemented for various use cases within the company, such as securing internal web applications like Jira or iManage, enabling SSH connections through Cloudflare Access, and providing a single location for users to launch any application they should be able to reach. The platform also offers per-request logging, allowing administrators to capture every request made to internal resources without any code changes. During the COVID-19 pandemic, Cloudflare has extended its offer of free access to its Teams product for businesses of all sizes until September 1, 2020. The company aims to help organizations maintain productivity and security while employees work remotely.

The URL (Uniform Resource Locator) is a reference to a resource on the Internet, such as an HTML file or image. It was first proposed by Tim Berners-Lee in 1994 and has since become the standard way of addressing resources on the web. A typical URL consists of several components: 1. Scheme: This is the protocol used to access the resource, such as HTTP (Hypertext Transfer Protocol) or FTP (File Transfer Protocol). It is followed by a colon and two slashes (://). 2. Host: This is the name or IP address of the server where the resource is located. For example, www.example.com. 3. Port: This is an optional component that specifies the port number on which the server is listening for requests. If omitted, it defaults to 80 for HTTP and 21 for FTP. 4. Path: This is the path to the specific resource within the host's file system. It starts with a forward slash (/) and can include multiple directory levels separated by more slashes. 5. Query parameters: These are optional key-value pairs that provide additional information about how to process the request. They are preceded by a question mark (?) and each parameter is separated from the next by an ampersand (&). 6. Fragment identifier: This is an optional component that specifies a particular part of the resource to which the browser should navigate when the URL is clicked. It is preceded by a hash symbol (#). The URL was designed to be human-readable and machine-processable, making it easy for both users and programs to locate resources on the web.

The Pwned Passwords API, a part of Troy Hunt's Have I Been Pwned service, has introduced padding in its responses to protect against potential attack vectors that use passive analysis of response sizes. By passing the "Add-Padding" header with a value of "true", users can request padded API responses. The padding consists of randomly generated hash suffixes with usage count set to "0". This feature is expected to be mandatory in the future, once clients have had time to update their implementations.

Juan M. Rodriguez has joined Cloudflare as its first Chief Information Officer, aiming to help scale the company during this new phase of business. He was impressed by Cloudflare's speed of innovation and transparency, as well as their commitment to doing the right thing with integrity and building trust with customers and partners. The company's focus on a diverse team and its mission to build a better internet also motivated Rodriguez to join. As CIO, he will help maintain Cloudflare's culture of innovation while ensuring compliance as a public company.

The article discusses the increasing reliance on JavaScript and third-party code in modern web architecture, which opens up potential vulnerabilities due to unmanaged integrations. It mentions that only 2% of Alexa 1000 global websites deploy client-side security measures against attacks like Magecart, XSS, credit card skimming, session redirects and website defacement. The article then introduces Cloudflare Workers as a solution to improve security and performance of web applications. It explains how Tala has integrated its "web module" capabilities into Cloudflare's service Worker platform for serverless instantaneous deployment. Tala automates the activation of standards-based, browser-native security controls without impacting website performance or user experience. The integration with Cloudflare Workers allows customers to activate enterprise-grade website security quickly and efficiently from Cloudflare's edge locations worldwide.

The Resource Public Key Infrastructure (RPKI) to Router Protocol, or RTR Protocol, is an essential part of securing the global routing table that makes for a safer internet. It communicates between a validator and device in charge of allowing or rejecting routes in its table. The RTR protocol allows heavy cryptographic processing to be done outside of routers, which have better things to do with their control plane processor like running BGP convergence algorithms or other routing-related tasks. Cloudflare has released an open-source software package called rpki-rtr-client that performs the router portion of the RTR protocol and can help test existing code bases. The future of RTR may include additional functions while staying lightweight, as it is expected to progress alongside other draft works in the IETF proposal for RTR version 2.

The author discusses their experience with using Bloom filters to remove duplicate lines from a large dataset containing 1 billion lines. They initially chose Bloom filters due to their ability to handle probabilistic data structures and the fact that they only needed to remove duplicates, not sort the lines. However, they encountered performance issues when the Bloom filter did not fit into L3 cache, leading to random memory accesses which are very costly. To address this issue, the author switched to using a simple hash table with linear probing, resulting in better memory access patterns and improved performance. They also learned that modern CPUs are really good at sequential memory access but struggle with random memory access, and that advanced data structures like Bloom filters can be great as long as they fit into L3 cache. In conclusion, the author emphasizes the importance of optimizing for reduced number of loads over optimizing memory usage when working with large datasets not fitting into L3 cache, and highlights the power of profiling tools to identify performance bottlenecks.

Cloudflare has deployed AMD's EPYC processors in its tenth generation server platform, marking the first time it is not using Intel components for any major server components. The Gen X server processes as much as 36% more requests while costing substantially less than its predecessor, Gen 9. It also enables a significant decrease in L3 cache miss rate and NGINX p99 latency. The AMD EPYC 7642 processor was selected after benchmarking various CPUs for key workloads such as cryptography, compression, regular expressions, and LuaJIT. Gen X server performance is further enhanced by partnering with AMD to tune the processor, achieving an additional 6% performance boost. Additionally, Cloudflare uses Secure Memory Encryption (SME) within the System on a Chip architecture of the AMD EPYC line for improved security without significant decrease in performance.