November 2020 Summaries

In this blog post, the authors discuss how they made their infrastructure DNS zone more reliable by using multiple primary nameservers to leverage their own DNS product running on their edge as well as a third-party DNS provider. They set up their infrastructure zone with multiple primary nameservers and added three nameservers from an additional provider to their zone at their registrar. This setup is similar to using a primary and secondary server, but the main difference is that the nameservers operate independently from one another. The authors also automated the process of maintaining their infrastructure DNS zone by using OctoDNS, which provides a set of tools that make it easy to manage DNS records across multiple providers. They plan to further automate the process in the future by reducing the need for approvals when adding new records.

This week on the Cloudflare blog, various topics were covered including improving website performance and search rankings with Cloudflare, benchmarking new server hardware at the Cloudflare Core, moving Quicksilver into production, building Black Friday e-commerce experiences with JAMstack and Cloudflare Workers, a Byzantine failure in the real world, and an overview of ASICs at the Edge. These articles provide insights into how Google Search will prioritize results based on web performance metrics, the efficiency improvements from refreshing hardware at the Cloudflare Core, the journey to production for Quicksilver data store, using Cloudflare Workers for e-commerce experiences, and understanding ASICs in Cloudflare's global network.

Cloudflare's global network relies heavily on Application Specific Integrated Circuits (ASICs) to efficiently manage traffic across over 200 cities in more than 100 countries. Unlike CPUs or GPUs, ASICs are designed for specific tasks, providing high efficiency, reduced power consumption, and lower failure rates, making them ideal for networking equipment where new technologies emerge slowly. ASICs facilitate critical networking functions like packet switching, routing, and firewall operations using specialized memory types like TCAM for rapid data retrieval. The networking ASIC market includes major players like Broadcom, Intel, Mellanox, Juniper, and Cisco, each offering unique solutions with varying throughput, buffer sizes, programmability, and feature sets. Broadcom dominates with its Tomahawk, StrataDNX, and Trident series, while Intel's Barefoot Tofino represents a paradigm shift with its programmable architecture. Mellanox's Spectrum ASICs offer high-speed switching capabilities, and Juniper's Trio and Express families provide robust routing solutions. Cisco's ASICs, including Bigsur and Tahoe, support their extensive network equipment portfolio. These technological innovations enable Cloudflare to optimize its network infrastructure, ensuring high-performance, scalable, and reliable services.

On November 2, 2020, Cloudflare experienced an incident that impacted the availability of its API and dashboard for six hours and 33 minutes. The issue was caused by a Byzantine fault, which led to a cascading series of events involving partial switch failure, etcd errors, promotion of new primary databases, and overloaded authentication databases. Despite having redundancy in each system, the combination of degraded states made it difficult to model and anticipate the chain of events that transpired. The incident led Cloudflare to revisit its configuration parameters for auto-remediation processes and prompted further research into Byzantine Fault Tolerance (BFT) consensus protocols.

The article discusses how Cloudflare's serverless platform, Workers, can be used for building an open-source e-commerce app. It highlights the benefits of using Workers as a static site hosting platform, API server, and webhook consumer within a single codebase. The author also explains how to integrate Workers with Stripe for payment processing and Sanity.io for data management. The article concludes by encouraging readers to explore the open-source project on GitHub and build their own e-commerce applications using Workers.

The text discusses the process of migrating from Kyoto Tycoon, an existing key-value (KV) data store, to Quicksilver, a new distributed KV data store at Cloudflare. This migration was challenging due to the need for zero downtime and seamless integration with existing systems. The team built QSKTBridge, a bridge service that replicated from Kyoto Tycoon and wrote batched changes every 500ms. They also gradually phased out reads and writes to Kyoto Tycoon while ensuring Quicksilver's performance was healthy before moving on to the next data center. The team faced several issues during this migration, such as replication saturation due to I/O contention, high write amplification causing increased pressure on SSDs, and challenges in configuring topology dynamically. They also discovered that removing Kyoto Tycoon from the edge took around a year, and removing it from the core was even harder. Despite these challenges, the team successfully migrated all Cloudflare services to Quicksilver, making significant improvements in performance and reliability. They are now working on a sharded version of Quicksilver to address storage size problems and sustain their growing product and customer bases. Overall, this migration process provided valuable insights into improving the KV store's design for future needs.

Cloudflare has redesigned its Core server fleet hardware to improve efficiency and lower complexity. The company designed two major server variants: Core Compute 2020, an AMD-based server for analytics and general-purpose compute paired with solid-state storage drives; and Core Storage 2020, an Intel-based server with twelve spinning disks to run database workloads. The new servers leverage learnings from the Gen X edge servers and have seen significant performance improvements in both synthetic benchmarking outside production and in the production workloads tested. This will allow Cloudflare to run the same workloads on fewer servers, saving CapEx costs and data center rack space.

Cloudflare is working closely with Google to improve web performance and page experience data for better search ranking results. The Core Web Vitals metrics include Largest Contentful Paint (LCP), First Input Delay (FID), and Cumulative Layout Shift (CLS). These metrics can be improved using Cloudflare's performance products, such as caching, Smart Routing, content optimization, and Workers. Google's recent announcement emphasizes the importance of running fast, accessible websites, and using Cloudflare's tools directly improves page experience and Core Web Vitals measurements, leading to higher search rankings.

Cloudflare has introduced a new feature for its Argo Tunnel service that allows users to expose unlimited local services through one instance of cloudflared. This update aims to address the pain point experienced by larger customers who previously had to manage multiple instances of cloudflared, leading to inefficiencies and complexities. The Named Tunnels feature enables traffic routing into a tunnel using DNS or Cloudflare Load Balancer records, while ingress rules map traffic from internet URLs to local services. This update also includes per-rule configuration options for different service requirements.

DDoS attacks are increasing in frequency and sophistication, with a 4x increase in network layer attacks compared to pre-COVID levels. The majority of these attacks are under 500 Mbps and 1 Mpps, but can still cause service disruptions. Ransom-driven DDoS (RDDoS) attacks are on the rise, with groups claiming to be Fancy Bear, Cozy Bear, and Lazarus Group extorting organizations worldwide. The number of L3/4 DDoS attacks observed by Cloudflare increased substantially in Q3, with a majority of attacks being under 1 hour in duration. Smaller attacks are becoming more common, potentially serving as smokescreens for other cyberattacks or as proof and warning to target organizations. Attack vectors such as SYN floods, RST floods, UDP floods, mDNS, Memcached, and Jenkins attacks have seen an increase in Q3. The United States observed the most number of L3/4 DDoS attacks, followed by Germany and Australia. Cloudflare's mission is to help build a better Internet, which grounds their approach to DDoS protection.

The text discusses the increasing vulnerabilities of firmware and hardware, particularly in the server industry. It highlights how secure boot technologies like Microsoft keys have been used to combat firmware-level attacks but questions their effectiveness in protecting the entire boot process. The article then delves into the boot process and explains how Cloudflare uses AMD's Platform Secure Boot (PSB) as a Hardware Root of Trust for its Gen X servers, ensuring the integrity and authenticity of System ROM image before execution. It also covers the build process, enabling PSB in the field, testing, and naming convention for signed images. The conclusion emphasizes the importance of hardware root of trust for code signing critical boot entities to ensure server hardware and software integrity through cryptographic means.

On November 16, 2020, Greg McKeon announced a Free Tier for Workers KV on the Cloudflare platform, allowing developers to store key-value data and make it globally accessible. The new tier includes 100,000 read operations and 1,000 each of write, list, and delete operations per day, with a maximum total storage size of 1 GB. Additionally, the maximum value size in Workers KV has been raised from 10 MB to 25 MB. These updates aim to expand the use cases for Workers KV and move towards providing global storage solutions for applications deployed across multiple data centers.

Cloudflare's Customer Success Managers provide continuous strategic and technical guidance through interactive workshops, account reviews, tuning sessions, and regular product updates. The company's product development and design teams are constantly working on new features and updates based on customer feedback. As part of the Premium Success offering, customers can be introduced to Product Managers for in-depth conversations about Cloudflare solutions and their application to business goals. Through machine learning and behavioral analysis, Cloudflare is able to ship new products to ensure security without impacting performance. Customer Success Managers aim to become trusted advisors by taking a genuine interest in customers' priorities and helping them reach desired goals. Collaboration and relationship building lead to secure platform consolidation and trust in those who manage it.

Researchers from UC Riverside and Tsinghua University have announced a new attack against the Domain Name System (DNS) called SAD DNS (Side channel AttackeD DNS). This attack leverages recent features of the networking stack in modern operating systems to allow attackers to revive a classic attack category: DNS cache poisoning. The researchers contacted Cloudflare and other major DNS providers, and 1.1.1.1 Public Resolver is no longer vulnerable to this attack. This post explains what the vulnerability was, how it relates to previous attacks of this sort, what mitigation measures have been taken to protect users, and future directions the industry should consider to prevent this class of attacks from being a problem in the future.

On November 13, 2020, Terin Stock announced the release of origin-ca-issuer, an extension to cert-manager that integrates with Cloudflare Origin CA for easy creation and renewal of certificates. This integration simplifies managing TLS certificates and keys within Kubernetes by providing Kubernetes resources that manage the lifecycle of a certificate. The origin-ca-issuer supports Ingress controllers, allowing automatic management of Certificate resources based on Ingress resources. It also includes an external cert-manager issuer for specialized Kubernetes controllers. Up-to-date documentation and installation instructions can be found in their GitHub repository, with contributions and feedback welcome.

In this blog post, Chad Toerien shares his experience of celebrating Black History Month at Cloudflare and how the company embraces diversity and inclusion through Employee Resource Groups (ERGs). As a South African working in the London office, he led the UK's Black History Month celebration in 2020. The event highlighted past triumphs, present contributions, and future rising stars of people with African and Caribbean heritage. Cloudflare used its infrastructure and services like Cloudflare for Teams and Cloudflare TV to celebrate the month remotely. Toerien emphasizes the importance of celebrating Black History beyond a single month and hopes to leave a legacy of hope by improving diversity and inclusion within the company.

Tech industry leaders discussed workplace evolution during a series of fireside chats commemorating Cloudflare's 10th birthday. Topics covered included the competition for talent, harnessing the benefits of remote work, capturing the serendipity of in-person meetings, securing a hybrid work environment, and digital transformation opportunities. The discussions highlighted the importance of flexibility, intentionality, and technology advancements to adapt to the changing work landscape.

Brotli is a state-of-the-art lossless compression format supported by all major browsers, offering better compression ratios than gzip. One of its key features is the built-in static dictionary that contains various strings in multiple languages and supports multiple transformations to increase versatility. The open-source brotli library has 11 predefined quality levels for the encoder, with higher levels demanding more CPU for better compression ratios. The Cloudflare team improved on the limited use of the static dictionary approach by adding optimizations that enhance compression at levels 5 through 9 without significantly impacting performance. They achieved this by using a larger, specialized subset of the dictionary and employing heuristics to improve the compression ratio. The resulting reduced dictionary is generated based on statistics about matches in representative content data sets for HTML, CSS, and JavaScript files. The improved dictionary approach allows for better compression of web content without requiring higher CPU usage compared to using a higher compression level. This enhancement benefits small files more significantly than large ones, making it an effective solution for optimizing web traffic.

On September 8th, Gartner published its latest Market Guide for Email Security, highlighting the need for Integrated Email Security Solutions (IESS) due to the increasing phishing attacks and enterprise migration to cloud email providers. Area 1 Security was recognized as a Representative Vendor for IESS, offering an attractive SEG replacement with its quick and easy deployment capabilities. The Market Guide also emphasizes differentiating capabilities for next-generation email security products such as network sandboxing, content disarm and reconstruction, URL rewriting, display name spoof detection, domain-based message authentication, lookalike domain detection, anomaly detection, graymail handling, data protection, post-delivery protection, M-SOAR integration, and more.

Boris Lecoeur has joined Cloudflare as its Head of France to expand the company's growing customer base in the country. This marks the fourth Cloudflare office in the EMEA region, alongside London, Munich, and Lisbon. Lecoeur brings over 20 years of experience in technology and cloud solutions, having previously opened the Amazon Web Services office in France. He is impressed by Cloudflare's growth, global network, innovative technology, and company culture. In France, Cloudflare has thousands of customers across various industries and organizations, including major enterprises like L'Oréal and Criteo. The company aims to help French organizations grow by providing secure, fast, and reliable Internet services.

A spear-phishing campaign is threatening targets with false claims of employment termination due to economic impacts from the global pandemic. The goal of the attacker is to intimidate employees into clicking on a link that will ultimately lead to Bazar or Buer malware infections by way of Trickbot. This campaign employs a number of lures that threaten job security to coerce targets into clicking on the provided URL. The phishing messages are very simple and appear to originate from persons of authority within the targeted company, focusing on either employment termination or customer complaints. The use of common cloud-based hosting services allows the attacker to circumvent URL scanning techniques, as well as enables them to easily create new malicious links in the event that their URLs are identified as phishing pages.

Cloudflare has expanded its Network Analytics dashboard to Spectrum customers on the Enterprise plan, providing enhanced visibility into Internet traffic patterns and DDoS attacks in Layers 3 and 4 of the OSI model. The expansion also introduces two major improvements for easier reporting and investigation: DDoS Insights & Trends and Group-by Filtering for grouping-based traffic analysis. These enhancements enable Spectrum customers to view detailed traffic reports on DDoS attacks, receive real-time alerts during service attacks, and better understand their security landscape.

The Internet's security is being improved through Routing Public Key Infrastructure (RPKI), which provides a source of truth for IP addresses and Autonomous Systems in signed cryptographically signed records called Route Origin Objects. Since the launch of the website "isbgpsafeyet.com" to test if ISPs correctly perform BGP Origin Validation, there has been significant growth in RPKI deployment across all network operators. Currently, 25% of routes are signed and 20% of the Internet is doing origin validation, with major transit providers such as Cogent, GTT, Hurricane Electric, NTT, and Telia making downstream networks more secure without them having to deploy validation software. The growth of RPKI adoption is expected to continue, leading to a safer Border Gateway Protocol (BGP) routing system in the future.

Cloudflare uses ClickHouse extensively for its internal analytics workload, bot management, customer dashboards, and other systems. The largest cluster has over 100 nodes, with several others having at least three nodes each. They use the standard approach in ClickHouse schema design, with clusters holding shards and a node being a physical machine. To manage capacity planning, they periodically collect extensive information from system tables about operating processes. They previously used manual methods to predict disk usage but have now automated this process using Python and Facebook's Prophet for time-series forecasting. The metrics are stored in ClickHouse itself, with real and predicted data displayed on the same dashboard in Grafana. This automation has helped Cloudflare spot unexpected disk space issues across multiple clusters and provides a valuable tool for planning server purchases. The project was carried out by the Core SRE team to improve daily work and is now used by other teams within the company.

Tech industry leaders and thinkers gathered for a series of fireside chats to discuss various topics such as entrepreneurship, mental health, leveling the playing field, expecting the unexpected, and the future of video conferencing. Anu Hariharan from Y Combinator's Continuity Fund highlighted the global opportunity for entrepreneurs, predicting that at least 50% of startups will be headquartered outside the US in a decade. Karan Singh, co-founder and COO of Ginger, shared his vision of a world where mental health is never an obstacle and how virtual care can become more democratic and equitable. Jennifer Hyman, CEO and co-founder of Rent the Runway, expressed optimism that flexible work arrangements post-vaccine will help level the playing field for women in the workplace. Eric Schmidt, former CEO & Executive Chairman of Google, emphasized the importance of machine learning investments and how they will yield unexpected applications. Lastly, Zoom founder and CEO Eric Yuan discussed the limitations of video conferencing compared to face-to-face meetings but expressed confidence that technology advancements could eventually make online communication better than in-person interactions.