August 2020 Summaries

On August 31, 2020 at 12:00 PM, Connor Peshek introduced the nineteenth edition of The Serverlist. This publication provides updates on serverless technology, offers developer tutorials, encourages discussions among serverless developers, and highlights upcoming meetups and conferences. To receive The Serverlist directly in your email, sign up using the provided link while ensuring your privacy is respected.

On August 30, 2020, CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare's customers as well as other services across the internet. The incident began at 10:03 UTC with an increased number of errors reaching Cloudflare's customers' origin servers. Within seconds, Cloudflare's systems automatically rerouted traffic from CenturyLink/Level(3) to alternative network providers such as Cogent, NTT, GTT, Telia, and Tata. Despite these mitigations, some customers still experienced issues due to their hosting providers only having single-homed connectivity to the internet through CenturyLink/Level(3). The outage appeared to take all of CenturyLink/Level(3)'s network offline, causing a 3.5% drop in global traffic. While the exact cause is not yet known, one plausible scenario involves a bad Flowspec rule that prevented BGP from being announced. It took more than four hours for CenturyLink/Level(3) to resolve the issue.

Cloudflare has introduced support for asynchronous handlers in HTMLRewriter, which enables developers to perform tasks based on the content of an HTML document. This includes prefetching fonts and image assets or fetching user-specific content from a CMS. The new feature allows dynamic HTML injection based on the contents of the document without prior knowledge of what it contains. Developers can now customize HTML based on a particular user, feature flag, or even an integration with a CMS. Examples of projects using HTMLRewriter include elegant web scrapers, image optimization with lazy loading, localizing websites, extending Ghost blogs, and prefetching Google fonts.

A recent phishing campaign has been exploiting the high demand for face masks and forehead thermometers during the COVID-19 pandemic. The attacker sends email attachments infected with Agent Tesla malware, an advanced Remote Access Trojan (RAT), to various companies under the guise of a mask production business venture. These phishing messages are designed to appear authentic by impersonating real employees at various companies and including accurate mailing and contact details. The attachment is manipulated to make it appear legitimate, often as a PDF file, when in fact it's a compressed executable containing the malware. Once downloaded, victims may only see "Supplier-Face Mask Forehead Thermometer.pdf", which is the actual file name. To further reduce suspicion, the attacker uses a double extension to trick targets into thinking the file is a PDF when it's actually an executable.

Argo Tunnel is a tool that helps make origins accessible from the Internet in a secure and seamless manner. The development team at Cloudflare uses Argo Tunnel for managing their fleet of machines and sharing private sites from localhost. They initially used Docker compose, but later switched to Skaffold + Minikube, and finally settled on Tilt + Docker for Mac k8s as it offered live updates without having to rebuild containers. The team also needed a way to share their development environment with distributed members, which led them to use Argo Tunnel in combination with Cloudflare Access. This setup allowed the team to expose remote dev environments securely and authenticate users before accessing the dashboard service.

Cloudflare has joined the Global Network Initiative (GNI), one of the world's leading human rights organizations in the information and communications technology sector, as observers. The company aims to formalize its commitment to help respect human rights online. As part of this commitment, Cloudflare will incorporate human rights training and due diligence tools like human rights impact assessments into their decision-making processes throughout the company. They recognize that there are limits to what they can do as a company but are committed to more formally documenting their efforts to respect human rights under the UN Guiding Principles on Business and Human Rights, with support from GNI.

Cloudflare has released wrangler 1.11, taking its edge-based development environment, wrangler dev, to general availability (GA). Wrangler dev is the world's first edge-based development environment and allows developers to run applications from Cloudflare's network of 200 datacenters, as close as possible to their users. The tool provides a high fidelity, consistent experience for all developers without sacrificing the speedy feedback loop of a local development environment. It also supports live reloading, console logs directly in the terminal, and testing cache APIs and KVs. Future plans include providing an easy testing framework for Workers and allowing customers to observe their Workers' behavior in production.

Joshua Johnson, an intern on the Workers Developer Productivity team at Cloudflare, has introduced "wrangler login," a new feature designed to improve the user experience when getting started with Wrangler. Wrangler is an open-source CLI tool used for managing Cloudflare's serverless platform, Workers. The main focus of Johnson's internship was enhancing the ease of use when starting with Wrangler. Previously, users needed to create an API token and authenticate it manually, which could be confusing and time-consuming. With "wrangler login," users can now log in using their Cloudflare credentials directly from the terminal, making the process much smoother. Additionally, security measures such as asymmetric RSA encryption have been implemented to protect user tokens. The new feature is available in the 1.11.0 release of Wrangler and aims to make getting started with Workers more seamless and enjoyable for users.

Cloudflare recently improved upload speeds across their network for clients using HTTP/2. They discovered that HTTP/2 upload speeds were sometimes much slower than HTTP/1.1 when the user had high available upload bandwidth. This issue was related to receiver flow control, which differs between HTTP/1.1 and HTTP/2. To address this problem, they implemented autotuning of HTTP/2 request body buffer size based on network conditions. The new feature is now fully deployed in the Cloudflare edge, benefiting customers with improved upload performance for all HTTP/2 connections.

Ryan Jacobs, an intern at Cloudflare, was tasked with improving the usability of Spectrum, a layer 4 reverse proxy, for internal customers. The existing process required manual configuration updates and was time-consuming. To solve this problem, Jacobs utilized the Addressing API, which manages IP prefixes and delegations within Cloudflare. By integrating the Addressing API with Spectrum's configuration service, he allowed particular customers to use Cloudflare managed addresses as origins for their Spectrum apps. This approach also mitigated potential issues such as cycles or connections to non-public Internet IP ranges. The integration required making an additional HTTP request on the critical path of every create app request in the Spectrum configuration service, but performance testing showed minimal impact. Jacobs' work contributed to improving the overall efficiency and usability of Cloudflare's internal services.

In August 2020, Twitter faced a security breach where attackers compromised an internal administrative panel to take over high-profile accounts. The attackers used spear phishing to steal credentials and access the Twitter control plane. This incident highlights the risk of using administrative panels as they can be targeted by phishing attacks. To prevent such incidents, Cloudflare has implemented several safeguards including hard keys for team members who need to access their admin panel. They also use Cloudflare Access to enforce the use of hard keys and limit access to specific users in permission groups. This zero-trust approach ensures that only authorized personnel can access sensitive systems and data, reducing the risk of security breaches.

Deploy Buttons is a new feature by Cloudflare that allows developers to deploy projects on the Workers Platform without setting up a local development environment. This makes it easier and faster for beginners to get started with serverless computing, as well as share their projects with others. The process involves clicking a "Deploy to Workers Button" on the project's README, connecting Github and Cloudflare accounts, and then deploying the project in three steps. Developers can also create their own Deploy Buttons by adding GitHub Actions workflow, support for CF_API_TOKEN and CF_ACCOUNT_ID, and Markdown code to their project's README.

Secondary DNS is a backup system for primary authoritative DNS servers, ensuring redundancy and synchronization of records across multiple servers. It can be used in various ways such as passive or active backups, and with hidden primaries. Secondary DNS Override enhances this by allowing users to proxy A/AAAA/CNAME records through Cloudflare's network while maintaining the primary server's control over record changes. This feature is particularly useful for those who want to leverage Cloudflare's security and performance services without transferring all their zones to Cloudflare DNS as a primary provider. Secondary DNS Override is currently available on the Enterprise plan, with additional documentation provided in support articles.

Cloudflare has announced significant updates to its Workers Docs, including reorganization of content into four categories for easier navigation, improved formatting with more structure and detail, redesigned pages with new UI components, accurate search functionality, and the addition of dark mode. The updated documentation aims to provide a better developer experience and make it more approachable for beginners. These improvements will also be applied to other Cloudflare products' documentation in the future.

Area 1 Security has identified a new Microsoft phishing campaign, dubbed "Summer Bonus," targeting frontline workers during the pandemic. The attackers leverage social engineering techniques and flaws in legacy email solutions to deceive recipients into divulging their Microsoft credentials. Two variants of this campaign have been observed: one using Microsoft SharePoint notifications and another spoofing Microsoft Planner emails. Both phishing attempts lead victims to a fake login portal hosted on cloud-based platforms like Google AppSpot, Azure, and Amazon Web Services (AWS). The attackers use multiple cloud services throughout the attack process to evade detection.

Cloudflare is working to ensure the security and reliability of the internet for the upcoming 2020 U.S. elections, as more activities shift online due to COVID-19. The company has observed an increase in DDoS attacks from April to June 2020 and is tracking trends and vulnerabilities to better understand threats against critical election players. Cloudflare's Athenian Project provides security protections for state and local government election websites, while the Cloudflare for Campaigns initiative offers free security protection to political campaigns. The company has seen a significant increase in traffic to election websites and is working with governments under the Athenian Project to prepare for elections by identifying threats and protecting against malicious attacks.

On August 14, 2020, Harnish Kanani joined Cloudflare as its Chief Customer Officer. The company has experienced significant growth since its launch a decade ago and now serves nearly 3 million customers. As the Chief Customer Officer, Kanani aims to ensure that all Internet properties are safe, reliable, and fast. He brings extensive experience in customer success from previous roles at companies like Palo Alto Networks, RedLock Security, and CipherCloud. The Cloudflare Customer Success Team is dedicated to serving customers' business needs with a proactive approach, offering various services such as adoption workshops, best practices sessions, and digital assets. They are constantly hiring more team members globally and working on enhancing their service models for better customer experiences.

On August 7, 2020, Cloudflare introduced Argo Tunnel RDP Bastion mode to simplify protecting Remote Desktop Protocol (RDP) connections at scale. This new feature functions as a jump-host and can reside on a single node in the network, eliminating deployment headaches. The adoption of remote work increased significantly due to the COVID-19 pandemic, leading to an increase in RDP usage. However, many organizations have misconfigured RDP, exposing their systems to potential attacks. Cloudflare Access adds stronger authentication to RDP sessions by first locking down access and enforcing identity-based policies. The new bastion mode allows users to reach any machine through a single cloudflared instance, making it easier for organizations to protect internal resources.

In Q2 2020, the number of L3/4 DDoS attacks observed over Cloudflare's network doubled compared to that in Q1. The scale of the largest L3/4 DDoS attacks increased significantly, with some of the largest attacks ever recorded observed during this period. More attack vectors were deployed and attacks were more geographically distributed. In May 2020, there was a significant increase in the number and size of attacks, coinciding with the heightened global pandemic situation. Small attacks continued to dominate in numbers as big attacks got bigger in size. The majority of L3/4 DDoS attacks observed in Q2 peaked below 10 Gbps, but these 'small' attacks can still easily cause an outage or service degradation if not protected by a cloud-based DDoS mitigation service. In terms of duration, 83% of all attacks lasted between 30 to 60 minutes. The largest DDoS attack on Cloudflare's network in Q2 was detected and mitigated automatically, with no impact to performance due to the size and global coverage of their network. The United States received the most number of L3/4 DDoS attacks, followed by Germany, Canada, and Great Britain. In terms of total attack bytes mitigated, the United States led, but was followed by Hong Kong, Russia, Germany, and Colombia. 57% of all L3/4 DDoS attacks in Q2 were SYN floods, with a total of 39 different types of attack vectors used. Cloudflare's three-pronged protection approach includes Gatebot for detecting and mitigating globally distributed volumetric DDoS attacks, dosd for analyzing traffic and applying local mitigation rules when needed, and flowtrackd for detecting and mitigating the most randomized and sophisticated TCP-based DDoS attacks.

Cloudflare has introduced Cloudflare Network Interconnect (CNI), which allows customers to directly connect their branch and HQ locations with Cloudflare's network, bringing the company's full suite of network functions to their physical network edge. CNI offers security, reliability, and performance benefits compared to using the public Internet to connect to Cloudflare. It provides dedicated on-ramps for customers’ on-prem networks to get onto Cloudflare's highways. The service is available in three options: via a private network interconnect (PNI), over an Internet Exchange (IX), or through one of their interconnection platform partners.

Cloudflare has announced its Network Interconnection Partner Program to support its new CNI product. The program significantly increases interconnection options for customers, allowing them to connect privately and securely with Cloudflare in the location of their choice using the method or vendors they prefer. Virtual private interconnection is particularly attractive to customers with strict security postures and demanding performance requirements, without the added burden of ordering and managing additional physical cross connects and expanding their physical infrastructure. The program includes five launch partners: Console Connect by PCCW Global, Megaport, PacketFabric, Equinix ECX Fabric™, and Zayo Tranzact, spanning North America, Europe, Asia, Oceania, and Africa.

The author shares their experience as a Legal Intern at Cloudflare during the pandemic. They highlight the company's commitment to fostering communication, community, commingling, and collaboration. Despite working remotely, the intern was able to participate in meaningful projects and engage with team members through frequent virtual meetings. The internship provided valuable research opportunities and exposure to various legal topics that will inform their future coursework and career goals. Overall, the author expresses gratitude for the positive experience and believes Cloudflare's approach will make it a great place to work in the future.

The author discusses the increasing global focus on anti-racism following the murder of George Floyd. They emphasize that learning about anti-racism is not enough; it must be actively practiced in order to make a difference. The author highlights two major hurdles to overcome: fostering a sense of belonging and ensuring equal opportunities for career advancement. Employee Resource Groups (ERGs) can help with the first, but companies often struggle with the second. The author suggests that committing to Diversity, Equity, and Inclusion (DEI) every day is crucial in order to see meaningful change. This includes taking actions such as mentoring underrepresented individuals, scheduling coffee meetings with them, providing candid coaching, learning to value different approaches, watching educational segments, hosting listening circles, requesting workshops, suggesting ideas for building a more inclusive culture, and asking the opinion of someone from an under-represented group. The author believes that committing to DEI is important because it's easier to do nothing and maintain the status quo. However, having a company full of underrepresented people who are not being heard, seen, celebrated, or promoted will not lead to the desired outcomes. The author views DEI as an ongoing journey with milestones and achievements to celebrate along the way. They challenge readers to participate in a 21-Day DEI Challenge to help start this journey.