December 2020 Summaries

The development of Cron Triggers on Cloudflare Workers involved tackling a problem related to parsers and cron expression format, which varies between platforms and isn't standardized by a governing body. To solve this issue, the team wrote their backend service in Rust, leveraging its ecosystem for working across multiple languages. They also developed a parser called 'saffron', which supported all the extensions they wanted while leaving off other field extensions like seconds and years. The UI had different requirements, leading to the use of two JavaScript libraries for displaying information about given cron expressions. However, these libraries sometimes allowed users to add schedules that would be rejected by the API on submit, causing a non-optimal user experience. To address this issue, they synced the API and backend with another schedule runner entrypoint and added a validation endpoint to the API. They also made plans to remedy the situation by working on saffron to fill in for the one library holding them back in the UI. The quest for the one true parser continues as they aim to bring the number of parsers down from two to one, with one source of truth for the entire stack.

Celso Martinho shares an update on Cloudflare's Lisbon office, which has grown to include 74 talented employees in various fields such as Engineering, Security, and Infrastructure. Despite the challenges posed by the pandemic, the company continues to expand its presence in Portugal and aims to make Lisbon one of its top locations alongside San Francisco, Austin, Singapore, and London. With 28 open positions in Lisbon, Cloudflare encourages potential applicants to explore their resources such as blogs, GitHub projects, developers' website, community forum, Twitter account, and free product trials to learn more about the company and improve their chances of securing a position.

Cloudflare has introduced a new feature that allows users to set up their Access policies to require all user traffic to be filtered by Cloudflare Gateway, ensuring secure and centrally logged traffic for self-hosted and SaaS applications. This integration also enables the creation of rules determining which users can connect to specific parts of SaaS applications. The new feature addresses challenges such as malicious device activity before connecting to applications behind Access and data leaks from devices after reaching data behind Access. Additionally, it ensures that every connection to every application is logged, allowing organizations to audit employee usage of SaaS applications like Salesforce, Office 365, and Workday. The integration with Cloudflare Access prevents users from logging into a SaaS application without connecting through Gateway, ensuring data control rules are always enforced. This feature is available for Teams Standard or Enterprise plans, and documentation is provided to assist in getting started. A free plan allows testing of Gateway's DNS filtering and Access for up to 50 users at no cost.

In November 2020, Cloudflare detected and mitigated a unique ACK DDoS attack called "Beat" that targeted a Magic Transit customer. The attack's packet rate followed a wave-shaped pattern for over 19 hours, resembling an acoustics concept called beat. The attacker launched a flood of packets where the rate was determined by the equation of the beat wave. Cloudflare systems automatically detected and mitigated over 700 DDoS attacks that targeted this customer during the two days of the attack. Flowtrackd, a unidirectional TCP state tracking machine, played a crucial role in detecting and dropping the attack packets at Cloudflare's edge.

Cloudflare has acquired Linc, an automation platform that helps frontend developers collaborate and build powerful applications. The acquisition aims to accelerate the development of Cloudflare Pages, a fast, secure, and free way to build and host JAMstack sites. By integrating Linc's Frontend Application Bundles (FABs) with Cloudflare Workers, users will be able to host both frontend and backend components together at the edge of the Internet, close to their users. This integration will enable richer and more powerful full-stack applications while setting a new standard for web performance by delivering the speed of powerful servers near users.

Cloudflare has released HTTP filtering in its Gateway service, expanding protection beyond DNS threats to include those at the HTTP layer as well. This feature allows administrators to create policies based on a user's identity and correlate that identity to activity in the Gateway HTTP logs. The integration between an organization's chosen identity provider (IdP) and Cloudflare Access enables this capability, allowing customers to integrate one or more providers at the same time. Additionally, IT administrators can use the Gateway activity logs to filter the HTTP traffic logs for specific users and device IDs, providing visibility into Internet traffic across an organization. This feature is available to any Gateway Standard or Teams customer at no additional cost.

On December 16, the RedDrip Team from QiAnXin Technology released their findings regarding random subdomains associated with the SUNBURST malware in SolarWinds Orion. They discovered that DNS queries are created by combining a unique GUID (based on hashing of hostname and MAC address) with a payload, which is a custom base 32 encoding of the hostname. The team also found that long domains are split across multiple queries where the second half is much shorter and unlikely to include a '.'. This causes the decoder to ignore many of the recorded DGA domains. They provided Python code for encoding and decoding the queries, including identifying random characters inserted into the queries at regular character intervals. The team also identified that the first 15 bytes of the encoded query are a GUID used to associate multipart messages.

The text discusses the optimization journey of an image processing pipeline at Cloudflare. It delves into various algorithms and techniques used to improve the speed of the matching algorithm in their CSAM image scanning tool. Initially, a naive quadratic algorithm was employed, but it proved to be slow. SIMD instructions were then utilized for improvement, but the performance still fell short of expectations. The Vantage Point Tree algorithm was attempted, but due to the large dimensionality of the problem, it failed to narrow down the search effectively. The authors discovered that they didn't need to fetch all data from memory and could compute only a subset of dimensions for each hash. This led to the development of a smarter brute force algorithm, which was further optimized by transposing data for better AVX usage. The final algorithm achieved significant speed improvements, reducing query times from 55ms to just 0.73ms. The text also highlights the challenges faced in making technologies work at Cloudflare scale and emphasizes the need for continuous optimization efforts.

On December 17, 2020, Rita Kozlov announced Cloudflare Pages, a fast, secure and free way for developers to build and host JAMstack sites. The platform aims to simplify the process of deploying websites by integrating seamlessly with existing development workflows. It offers features such as automatic builds triggered by git commits and pushes, unique preview URLs for each commit, infinite staging environments, multi-user access for unlimited users, and modern web standards support. In addition to static sites, Cloudflare Pages will also integrate with the company's serverless platform, Cloudflare Workers, allowing developers to build full-blown dynamic applications using JavaScript. The announcement highlights a vision for how new applications will be built on the web in the future.

On December 13, FireEye reported a sophisticated supply chain attack using SolarWinds' Orion IT monitoring software. The malware was distributed as part of regular updates to Orion with a valid digital signature. It hides its network traffic using a multi-staged approach and determines its command and control (C2) server using a domain generation algorithm (DGA). Analyzing DNS query traffic through Cloudflare's 1.1.1.1 resolver, a spike in traffic to avsvmcloud[.]com was observed starting in April 2020. The attackers added more unique subdomains over time and the geographic distribution of queries changed throughout the course of the attack. Cloudflare Gateway customers can block these threats and investigate DNS query logs for related malicious domains.

Mona Hadidi, a Strategic Customer Success Manager at Cloudflare, shares how the company incorporates customer feedback into its product roadmap. The process involves Customer Success Managers, Solutions Engineers, and Account Executives converting customer feedback into feature requests that are then routed to Cloudflare's Product and Engineering teams. These teams evaluate the requests based on factors such as urgency, volume of requests, partnerships, and strategic direction from leadership. The Secondary DNS Override feature is an example of a product developed in response to customer feedback. Maintaining strong relationships with customers and ensuring they derive maximum value from Cloudflare's products is crucial for the company.

Cloudflare has launched a week-long event called "Privacy & Compliance Week" aimed at highlighting its commitment to providing built-in data protection and privacy throughout its global network and for every product in its portfolio. The company announced the launch of the Data Localization Suite, which helps businesses get the performance and security benefits of Cloudflare's global network while making it easy to set rules and controls about where their data is stored and protected. Other initiatives include improving Internet protocols with respect to privacy, deprecating the __cfduid cookie, launching a new free Web Analytics product for everyone, announcing Workplace Records for Cloudflare for Teams, securing the post-quantum world, encrypting WAF payloads with Hybrid Public Key Encryption (HPKE), and supporting jurisdictional restrictions for Durable Objects.

Cloudflare has introduced Jurisdictional Restrictions for Durable Objects, ensuring that data is stored and processed only within a specific geographical region. This feature simplifies the process of building serverless, stateful applications that comply with today's regulations and can adapt to new policies as they emerge. Developers can easily set restrictions at an object level without worrying about infrastructure management or overhead. Currently, only EU jurisdiction is supported, but more will be added based on demand.

Area 1 Security researchers have identified a Microsoft SharePoint phishing campaign that leverages new COVID-19 restrictions to steal victims' login information. The attacker targets upper-level management and executives, using Virtual Private Servers (VPS) to send their phishing messages. The malicious links direct users to spoofed Microsoft login pages hosted on various cloud computing platforms. This campaign is difficult to detect due to its use of VPS and leading email service providers, as well as abuse of multiple cloud services throughout several stages of the attack.

Quantum computing is advancing rapidly due to the efforts of academic institutions and companies like IBM, Microsoft, Google, and Honeywell. This technology could potentially break modern cryptographic algorithms, posing a threat to internet security. Researchers are working on developing quantum-resistant cryptography solutions to protect data from future quantum computers. The National Institute of Standards and Technology (NIST) is leading the charge in defining post-quantum cryptography algorithms to replace RSA and ECC. Cloudflare has been actively testing and implementing these new algorithms, with plans to offer post-quantum cipher suites to customers as standards emerge.

Cloudflare has introduced encrypted Web Application Firewall (WAF) matched payload logging to provide better visibility into security events while ensuring user privacy. The feature logs only the malicious components of requests and encrypts them using a customer-provided key, preventing access by Cloudflare employees. It uses Hybrid Public Key Encryption (HPKE), an emerging standard developed in part by Cloudflare. This feature will be available to Enterprise customers in the coming months.

Cloudflare has implemented policies to address concerns about access to personal data, focusing on the most privacy-protective legal standards when conflicts arise between different countries' laws. The company maintains commitments such as never installing law enforcement software or equipment on its network and not modifying customer content at the request of any third party. Cloudflare believes in providing notice to customers when their data is requested by government entities, and it has challenged non-disclosure orders without end dates. The company's approach emphasizes being principled, transparent, privacy-respecting, due process-oriented, and customer-notifying.

Cloudflare has introduced a new beta feature called Workplace Records for its Cloudflare for Teams product. This feature uses Access and Gateway logs to provide the state and country from which employees are working, helping finance, legal, and HR departments determine where payroll taxes are due and providing a record to defend those decisions. The system respects team member privacy by only logging resolution within a taxable jurisdiction, not a specific address. It also helps enforce regional compliance for company policies and can block users from connecting to applications from specific regions. Workplace Records is available as a report for free to all Cloudflare for Teams customers.

Cloudflare, a company focused on improving security and privacy in its products and services, has obtained several security certifications and reports to provide transparency into its security processes, controls, and practices. These include SOC-2 Type II / SOC 3, ISO 27001:2013, PCI DSS, and HIPAA/HITECH Act compliance. Additionally, the company has established a Security Engagement Team to work with customers' security and compliance functions, addressing their concerns and ensuring that Cloudflare meets their regulatory and compliance needs. The ultimate goal is to continuously improve the security and privacy of its products and services based on customer feedback.

Cloudflare has introduced a new free Web Analytics tool that can be deployed by adding an HTML snippet to your website, making it easier for users to understand visitor behavior without compromising user privacy. The analytics service is designed to provide essential web analytics to everyone with a website, offering features such as page views, visits, top pages, countries, device types and referrers. It also supports global filters, zooming in on specific time periods, group by actions, and more. Unlike popular analytics vendors that track visitor behavior for ad revenue, Cloudflare's mission is to help build a better internet without compromising user privacy. The company plans to integrate the Web Analytics tool more closely with its other services and enhance it as a standalone product in the future.

Cloudflare has announced that it will deprecate the __cfduid cookie starting on 10 May 2021. The last __cfduid cookies will expire 30 days after that. This change is due to concerns about privacy and user data, as well as a desire to simplify web services. The primary use of the __cfduid cookie was for detecting bots on the web, which can disrupt service or compromise security. Cloudflare has been experimenting with alternative methods for bot detection that do not rely on collecting end-user IP addresses and will continue to work on ensuring quality in their bot management products after removing this cookie.

Encrypted Client Hello (ECH) is a new extension for Transport Layer Security (TLS) that aims to enhance the privacy of critical internet protocols by encrypting metadata during the handshake process. ECH protects sensitive information such as Server Name Indication (SNI), which can be used to infer details about client-server communication, and lays the groundwork for future security features and performance enhancements while minimizing their impact on user privacy. The development of ECH is a collaborative effort between academics and tech industry leaders, including Cloudflare, Fastly, Mozilla, and others. While not yet ready for large-scale deployment, ECH represents a significant upgrade to the TLS protocol, building upon emerging technologies like DNS-over-HTTPS.

Cloudflare, Apple, and Fastly have co-authored a new proposed DNS standard called Oblivious DNS over HTTPS (ODoH), which separates IP addresses from queries to prevent any single entity from seeing both at the same time. The protocol adds a layer of public key encryption and a network proxy between clients and DoH servers, ensuring that only the user has access to both the DNS messages and their own IP address simultaneously. ODoH is an emerging protocol being developed at the IETF and provides three main privacy guarantees: the target sees only the query and the proxy's IP address; the proxy has no visibility into the DNS messages, with no ability to identify, read, or modify either the query being sent by the client or the answer being returned by the target; and only the intended target can read the content of the query and produce a response. Cloudflare has made source code available for ODoH, allowing anyone to try out the protocol or run their own service.

Passwords are a significant security issue due to their vulnerability during transmission and storage. Even when encrypted, passwords can be leaked and broken with effort. OPAQUE is a cryptographic protocol that enables password usage without ever leaving the user's possession, thus ensuring greater security. It is an example of systems that allow password-based authentication while maintaining privacy. Although it requires significant changes to existing infrastructure, OPAQUE offers promising potential for enhancing password security in the meantime.

Cloudflare has announced several initiatives aimed at improving internet protocols for privacy. These include Encrypted Client Hello (ECH), Oblivious DNS-over-HTTPS (ODoH), and a superior password authentication protocol, OPAQUE. The company is working with standards bodies like the IETF and partners such as Mozilla, Google, and Equinix to design, deploy, and test these new privacy-preserving protocols at internet scale. These projects aim to protect users' online lives and digital footprints by addressing critical aspects of their online experiences.

Cloudflare has introduced the Data Localization Suite to help businesses manage where their data is stored and protected while benefiting from its global network's performance and security features. The suite, available as an add-on for Enterprise customers, enables users to set rules and controls at the edge about data localization. It addresses challenges faced by companies in maintaining data locality while ensuring global accessibility of their internet properties. The Data Localization Suite includes encryption tools, control over TLS private keys' locations, management of HTTPS request and response inspection, and Edge Log Delivery for logs and analytics. Additionally, Cloudflare Workers now supports Jurisdiction Restrictions for Durable Objects, allowing users to ensure that their applications store data within a specific jurisdiction.

The first phase of the internet lasted until the early 1990s, focusing on connectivity and growth without significant concerns about data security or privacy. In the second phase, beginning in the mid-1990s with the introduction of SSL, security became paramount, enabling e-commerce and secure online transactions. However, this security was often misunderstood as guaranteeing privacy, which it did not. The third phase, now underway, focuses on privacy as a core element alongside universal connectivity and security. Governments and citizens are demanding control over their data, leading to complex regulatory challenges for companies that have built businesses around aggregating user information. New tools and protocols are being developed to build a privacy-respecting internet and give companies the means to easily create privacy-respecting applications.

Cloudflare's Privacy & Compliance Week begins on December 7th, focusing on ways customers can ensure compliance with global privacy rules and laws. The company emphasizes trust as the foundation of its business, having made decisions that prioritize user privacy over potential profits. In 2013, Cloudflare sued the US government to challenge National Security Letters' infringement on due process principles. The company has also refused to participate in ad tech companies' data collection practices and has invested in technologies to keep users' private data secure. As part of this week's announcements, Cloudflare is deprecating its _cfduid cookie, giving users more control over data processing locations, and introducing the Data Localization Suite to help customers comply with increasingly complex local data processing requirements.

Rob Dinh's blog post delves into the role and management of Power Distribution Units (PDUs) in data centers, emphasizing their importance in maintaining power redundancy and uptime. The text explores the technical aspects of PDUs, including their use of Alternating Current (AC) power guided by Faraday's and Ohm's Laws, and the advantages of 3-phase PDUs over single-phase systems, such as increased power capacity and reduced operational costs. The post also explains the mechanics of 3-phase power generation and the significance of balancing power loads across different branches to prevent capacity overloads and ensure efficiency. The discussion includes practical insights into optimizing PDU usage, particularly in the context of Cloudflare's global network infrastructure, highlighting the company's efforts to manage diverse power standards while maintaining reliable hardware operations.

Tech industry leaders gathered at Cloudflare's 10th birthday event, discussing the importance of diversity and internet accessibility. Selina Tobaccowala emphasized the value of collecting diverse feedback through anonymized surveys. Bonita Stewart shared her mission to make technology accessible to African-American communities by providing immersive experiences in historically Black colleges. Erik Hersman explained how his company, BRCK, offers free Wi-Fi access in Kenya and Rwanda through advertising and microwork tasks. Shellye Archambeau discussed defying expectations as a leader in Silicon Valley and using a servant leadership approach. Pam Kostka encouraged being intentional about expanding networks to promote diversity and make it a strategic imperative within companies.