April 2023 Summaries

Researchers Pedro Umbelino and Marco Lux have discovered a new DDoS reflection/amplification attack vector, CVE-2023-29552, exploiting the Service Location Protocol (SLP). The protocol was designed for service discovery in local area networks but has no authentication method and is not meant to be exposed to the public internet. Despite its obsolescence, many commercial products still support SLP, with 35,000 Internet endpoints having their devices' SLP service exposed. UDP version of this protocol has an amplification factor of up to 2,200x. Cloudflare customers are already protected from these attacks through the company's automated DDoS protection system. Network operators should block UDP port 427 or use Cloudflare Magic Firewall rules to prevent exploitation and launching of such attacks.

Grant Bourzikas, a seasoned CSO with over 20 years of experience in financial and private sectors, has joined Cloudflare as Chief Security Officer. He is thrilled to be part of the innovative company and contribute to its mission of creating a secure, faster, and more reliable internet for everyone. His charter includes defending the company and protecting 20% of all websites from sophisticated threats while promoting a culture of innovation in the ever-evolving cybersecurity landscape. Bourzikas was impressed by Cloudflare's strong mission, direction, commitment to building a better Internet, their advanced network capabilities and cloud technology, impressive customer base, as well as its passionate and professional employees during his interview process.

The architecture of Bumblebee follows the Unix principle of "do one thing and do it well". Instead of adding functionalities directly to the main proxy application, smaller services are created which can be restarted without dropping connections. These smaller services improve development agility and system reliability. Unix Domain Sockets are used for inter-process communication in Bumblebee. Unix sockets allow passing file descriptors between different processes, a feature essential for graceful restarts and the overall architecture of Bumblebee. The hot-potato library is used for file descriptor passing. Graceful restart is implemented using tokio and file descriptor passing. When an upgrade request (usually SIGHUP) is received, all tasks are paused, waited to be fully paused, and sent to a new process. To implement graceful shutdown, which is necessary for graceful restart, tasks must be made cancellation-safe. The results of the paused tasks are collected into a JoinSet and then passed to the new process using file descriptor passing.

The Cloudflare team has developed a new approach to secure-by-design development that ensures the security of its products from the start. This process involves investing significant resources into developing secure systems, prioritizing features and tools that protect customers rather than expanding product features that enlarge the attack surface. By building secure products that are easy to adopt and require minimal ongoing customer oversight, Cloudflare is able to reduce the likelihood of being compromised. The company also employs techniques such as educating executive leadership on the importance of continual reinvestment in modern security standards, and running experiments to build credibility for its secure-by-design approach.

Memory organization and memory width has a slight effect on memory bandwidth performance. The difference is most obvious in write-heavy workloads than read-heavy workloads. But even in write-heavy workloads, the difference is less than 4% according to our benchmark tests. Memory modules with x4 width require twice the number of memory devices on the memory module, as compared to memory modules with x8 width of the same capacity. More memory devices would consume more power. According to Micron's measurement data, 2Rx8 32GB memory modules using 16Gb devices consume 31% less power than 2Rx4 32GB memory modules using 8Gb devices. The substantial power saving of using x8 memory modules may outweigh the slight bandwidth performance impact. Our Gen11 servers are configured with a mix of 2Rx4 and 2Rx8 DDR4 modules. For our future generations, we may consider using 2Rx8 memory where possible, in order to reduce overall system power consumption, with minimal impact to bandwidth performance.

During Easter week, traffic dropped significantly in several countries with predominantly Christian populations, including Spain and Italy in Europe; Brazil, Mexico, Argentina, Colombia and Chile in Latin America; Australia in Asia and Oceania; and the Philippines. For example, on Easter Sunday, April 9, traffic dropped by almost a third (~30%) in Spain, compared to the previous week. In Greece, which celebrates Orthodox Easter, Internet traffic was 26% lower on April 16 than in the previous week. This blog post highlights these trends and more using Cloudflare Radar tool or simply via social media accounts like Twitter (@CloudflareRadar) or Mastodon (https://cloudflare.com/product/mastodon) , where is?. The future of this technology will likely be used in various ways as time passes forward into the foreseeable future!.

Cloudflare Zaraz, a solution that enhances the speed and security of third-party tools on websites, has introduced a new feature called Zaraz Consent Manager. This feature allows website owners to easily gather and manage user consent preferences for various third-party tools that use cookies or tracking technologies. The Consent Manager uses an opt-in approach, where non-essential trackers are disabled by default until the user grants their consent. This new tool helps businesses navigate the evolving privacy landscape and maintain compliance with various regulations across different jurisdictions.

Cloudflare has introduced an "Internet quality" feature called Aggregated Internet Measurement (AIM), which measures how well a user's network is performing and what activities it supports. The score ranges from 0-100 and provides insights into streaming, gaming, and real-time communication experiences on the internet. Cloudflare has also made this data publicly available through their partnership with Measurement Lab (M-Lab) for end-users and network engineers alike to parse through network quality data across a variety of networks. Additionally, they have open-sourced their speed test client to provide more accurate measurements of internet performance. This new feature aims to help improve the overall internet experience for users by giving them better information about their network's capabilities.

A detailed study shows that the latency and not throughput determines the speed of an internet connection. Lowering latency results in significant reduction in page load times, irrespective of whether the internet connection is slow or fast. On the other hand, increasing bandwidth does not necessarily improve page load times. The correlation between lower latency and reduced page load time exists even when latency is already low. This understanding can help us build a more performant Internet.

Cloudflare has announced that its Security Service Edge (SSE) solution, Cloudflare One, has been recognized as a "Leader" in the 2023 Gartner Magic Quadrant for SSE. The company's commitment to providing comprehensive security and connectivity features at an affordable price point contributed to this recognition.

In Q1 2023, we identified and analyzed several significant Internet disruptions around the world. These incidents include shutdowns due to political unrest in Iran and Turkey, natural disasters like Cyclone Freddy impacting Malawi, Zambia, and Mozambique, ongoing cyberattacks on South Korean ISP LG Uplus, a network incident in the Republic of the Congo, and power outages resulting from Russian attacks on energy infrastructure in Ukraine. We also saw multiple service disruptions at Docomo Pacific due to reported cybersecurity incidents in Guam.

Cloudflare's new Network Analytics dashboard provides network operators with a real-time, in-depth view of their traffic patterns and security events across their entire global network. By analyzing these data points, teams can quickly identify potential issues or malicious attacks and respond appropriately to mitigate them before they cause disruptions to critical services. The Network Analytics dashboard also enables users to create custom reports tailored to their specific needs, helping decision makers better understand their threat landscape and security posture.

In Q1 2023, China became the top source country for DDoS attacks, while Russia remained in second place. The United States was the third most common source of such attacks. Meanwhile, Hong Kong replaced China as the number one target country, with the United States and France taking the second and third spots, respectively. The majority of DDoS attacks observed during this period were volumetric, which involve overwhelming a victim's network or server by saturating it with traffic. These types of attacks accounted for 57% of all incidents in Q1 2023. Additionally, application-layer attacks also continued to pose significant risks to businesses worldwide. In terms of attack vectors, DNS amplification/reflection attacks remained the most prevalent, followed by UDP-based and TCP-based attacks. Emerging threats such as SPSS-based DDoS attacks, GRE-based attacks, and DNS

Cloudflare has introduced Zone Holds, a new capability for enterprise customers that prevents multiple accounts from managing the same zone unless explicit permission is granted by the domain owner. This feature ensures that one account can contain a specific domain or subdomain, eliminating potential conflicts and security issues when different teams within an organization use Cloudflare. With Zone Holds, users will receive an error message if they attempt to add a held domain to their account, prompting them to contact the domain owner for approval. This feature is enabled by default for all enterprise zones and can be managed from the Zone Overview screen.

On April 4th, UK ISP Virgin Media experienced an internet outage affecting its subscribers. The outage was identified by Cloudflare Radar data showing a significant drop in Virgin Media traffic around midnight UTC and partial recovery at various intervals throughout the day. Virgin Media acknowledged both disruptions on their Twitter account and provided updates to affected customers, however, no root cause for the series of disruptions has been publicly released yet. Analysis of Border Gateway Protocol (BGP) activity suggests that the underlying cause may be BGP-related or related to problems with core network infrastructure.

Implementing a graceful restart for an application can be complex, but it is essential to ensure service continuity and minimize impact on users during updates or configuration changes. To achieve this in Rust, you need to use tokio asynchronous runtime library that provides tools for managing IO operations and concurrency. In addition, creating a Unix socket specifically for coordinating restarts can help detect failures earlier and prevent your systems from getting into an inconsistent state. However, there are still certain limitations when it comes to carrying existing state to a new process, which requires more advanced techniques such as hot code upgrades or other creative solutions.

An SSL/TLS session resumption can occur when a client reestablishes a connection with a server by using an existing SSL/TLS session ID, which allows for faster connection setup times compared to establishing a new session from scratch. This process is often used in high-latency networks or where minimizing latency is critical. However, this particular vulnerability was discovered due to the fact that when a client presents a certificate during an SSL/TLS handshake, Cloudflare stores the certificate chain on the TLS connection and checks the revocation status before passing it along for further processing (such as evaluating Firewall Rules). When sessions were resumed, code to store the client certificate chain in application data did not run, resulting in an empty certificate chain. This meant that Cloudflare was unable to check the revocation status when using session resumption, which could lead to a situation where traffic from revoked mTLS certificates is allowed through Firewall Rules. This issue only affected mutual TLS (mTLS) connections, where both client and server authenticate each other using SSL/TLS certificates. To address this vulnerability, Cloudflare initially disabled session resumption for all mTLS connections to the edge. Subsequently, a fix was developed that supports both session resumption and revocation by removing the requirement of depending on data stored within the TLS connection state. Instead, an API call is used to grant access to the leaf certificate in both session resumption and non-session resumption cases. This issue highlights the importance of proper implementation and testing of security features such as SSL/TLS revocation checking, especially when dealing with high-value targets or sensitive data. By quickly identifying and addressing this vulnerability, Cloudflare has taken steps to ensure the continued protection of its customers' data.

On November 2022, Cloudflare introduced Deployments for Workers, allowing users to track changes in their Worker instances. In response to user demands and developer confidence, rollbacks have been added as a feature. This enables developers to quickly revert to previous versions of a Worker in case of errors or issues with the latest deployment. Rollbacks can be executed via the dashboard or Wrangler, Cloudflare's command-line tool. The rollback feature also includes record-keeping by allowing users to enter reasons for rolling back a deployment. This helps provide more context about why the rollback was necessary. The rollback function enhances confidence in deploying Workers and encourages developers to make use of this new functionality.