July 2020 Summaries

On July 31, 2020 at 12:00 PM, Connor Peshek introduced the eighteenth edition of The Serverlist. This publication provides updates on serverless technology, offers developer tutorials, encourages discussions among serverless developers, and highlights upcoming meetups and conferences. To receive The Serverlist directly in your email, sign up using the provided link while ensuring your privacy is respected.

The text discusses the development experience with Workers, a platform that provides serverless computing services. It highlights how Workers improves TTFD (time to first dopamine) and emphasizes the importance of this metric in the developer's journey. The author talks about the challenges faced while developing applications on existing serverless platforms and how Workers aims to overcome these issues by providing a delightful development experience. The text then delves into the four stages of the developer's journey: getting started, iteration, release, and observe. It explains how Workers simplifies each stage, making it easier for developers to write, test, deploy, and monitor their applications. The platform offers features like fast feedback loop, sandboxed nature, and accuracy in local development through wrangler dev. Workers also ensures quick deployment times (less than five seconds) without cold starts, allowing developers to release code frequently. Additionally, it provides tools for observability, such as Workers Metrics for trend identification and `wrangler tail` for production log access. The author concludes by encouraging users to provide feedback on their development experience with Workers.

Cloudflare has introduced a solution to eliminate cold starts, which are the time it takes to load and execute a new copy of a serverless function for the first time. Unlike other serverless platforms that require users to choose between suffering from random increases in execution time or paying extra fees to keep their functions warm, Cloudflare Workers now offer zero cold starts. This is made possible by leveraging encryption protocols during the TLS handshake process and preloading resources during the waiting time before a request arrives. The new feature is available for all Workers customers without any additional fee or configuration change.

Cloudflare has announced the general availability of Bring Your Own IP (BYOIP) across its Layer 7 products, Spectrum, and Magic Transit services. BYOIP allows customers to use their own IP prefixes with Cloudflare's services. The announcement includes listing BYOIP on relevant product pages, developer documentation, and UI support for controlling prefixes. Customers choose BYOIP for various reasons such as already allow-listed IP prefixes or having end users pointed directly to their IPs via DNS. BYOIP customers receive network analytics on their prefixes and can use the service with Layer 7 products, Spectrum, and Magic Transit. The process requires a Letter of Authorization (LOA) and up-to-date Internet Routing Registry (IRR) records before Cloudflare can announce the customer's prefix. Delegations manage how the prefix can be used across multiple Cloudflare accounts for Layer 7 and Spectrum BYOIP prefixes.

Cloudflare Workers, a serverless computing platform, has an architecture designed to ensure security when running code written by third parties. The platform uses V8 isolates for secure execution environments, which prevent code from accessing memory outside the isolate even within the same process. This allows many thousands of guest apps to be hosted on every machine with minimal overhead. Workers also employ a "layer 2" sandbox using Linux namespaces and seccomp to prohibit all access to the filesystem and network, restricting communication to local Unix domain sockets. The platform's security architecture is an ongoing project, with continuous efforts to reduce the risk and impact of future vulnerabilities.

Cloudflare has announced support for Python, Scala, Kotlin, Reason and Dart on its Cloudflare Workers platform. This enables developers to build applications using their preferred languages starting today. The process of getting started is simple, requiring the installation of Wrangler followed by running generate for the template of the chosen language. The blog post also discusses how support for these languages was added and how users can add support for other languages in the future.

The text discusses the migration of legacy applications onto Cloudflare Workers, a serverless compute platform that offers zero-trust security. It highlights the benefits of such a move, including instant scaling, no need for separate credentials, and reduced operational costs. The author demonstrates this process using an example application written in Node.js. They explain how to use four key Cloudflare technologies - Serverless Compute through Workers, Robust Developer-focused Tooling via Wrangler, Zero-Trust security through Access, and Instant, Secure Origin Tunnels through Argo Tunnels - to simplify the migration process. The text also provides a step-by-step guide on how to use these technologies in practice.

Cloudflare has launched Workers Unbound, a new platform that extends CPU limits and enables developers to bring all their workloads onto it, regardless of how intensive they are. This makes it possible for developers to run heavy computation tasks on the Edge without having to choose between running fast, simple work or running complex algorithms in a centralized cloud with unlimited resources. Workers Unbound is built as a general-purpose computing platform and aims to be more compelling than traditional, centralized serverless platforms. The platform offers significant cost savings compared to AWS Lambda due to its lightweight nature and efficient architecture. Cloudflare is currently offering Workers Unbound in a private beta for select developers.

A recent study by Area 1 Security reveals that state and local election administrators are at varying stages of cybersecurity readiness, with many relying on rudimentary or non-standard technologies to protect themselves from phishing attacks. The majority (53.24%) have only basic controls in place, while fewer than 2 out of 10 (18.61%) election administrators have implemented advanced anti-phishing cybersecurity measures. Additionally, a surprising 5.42% rely on personal email accounts or technologies designed for personal use to conduct their duties. Area 1 Security recommends that election administrators end the use of Exim email servers and transition to cloud email infrastructure such as Google's GSuite or Microsoft's Office 365 in combination with a cloud email security solution. They also advise against using personal email technologies for election duties.

Matthew Prince of Cloudflare discusses the growth and evolution of their edge computing platform, Cloudflare Workers, over the past three years. He highlights that while speed was initially considered the most important attribute for a development platform, consistency, cost, ease of use, and compliance have proven to be more crucial. The company is set to announce several enhancements to the platform during Serverless Week, aimed at enabling developers to build more complex applications, lower serverless computing bills, make applications faster, and ensure security. Prince also presents his Hierarchy of Developers' Needs, which ranks these attributes in order of importance: Speed < Consistency < Cost < Ease of Use < Compliance. He predicts that regulatory compliance will be the most significant factor driving edge computing over the next three years.

In this blog post, Els Shek shares her experience as a Field Marketing and Events Manager at Cloudflare. She discusses her inspiration to join the company after reading a response from co-founder Michelle on Quora about what it's like to work there. Els details her first year at Cloudflare, including attending orientation in San Francisco, working with her team in Singapore, and participating in various events and projects across the Asia-Pacific region. She also highlights the company's focus on career planning, inclusivity, and employee well-being during the COVID-19 pandemic. Els concludes by expressing her excitement for the future and encourages others to consider joining Cloudflare.

Pablo Viera, a Latino immigrant from Argentina, shares his experience working at Cloudflare since 2018. Initially hired as the first Business Development Representative for Latin America, he has been promoted to Sales Executive for North America and appreciates the company's commitment to diversity and inclusion. He highlights various Employee Resource Groups (ERGs) created within the organization, such as Afroflare, Desiflare, Nativeflare, Latinflare, Proudflare, Soberflare, and Vetflare. Viera emphasizes that Cloudflare values diverse workforce representation and encourages under-represented groups to apply for positions within the company.

In order to provide a seamless user experience, it is essential that applications are available in multiple languages. This process of making an application ready for translation into different languages is known as internationalization (i18n). The following steps can be taken to achieve this: 1. **Extract all translatable strings**: All text displayed on the screen should be stored separately from the code, typically in a JSON file or similar format. This makes it easier to manage and update these strings without having to modify the application's source code. 2. **Ensure that all data is properly formatted for each locale**: Different countries have different conventions when it comes to formatting numbers, dates, times, currencies, etc. Therefore, any data displayed on the screen should be correctly localized based on the user's language and region settings. 3. **Translate all extracted strings into the desired languages**: This can either be done manually by human translators or automatically using machine translation services. In both cases, it is important to ensure that the translated text accurately conveys the intended meaning of the original English text. 4. **Integrate the translated strings back into the application code**: Once all strings have been translated, they need to be integrated back into the application so that users can see them when they use the app in their preferred language. 5. **Test and debug the localized version of the application**: After integrating the translated strings, it is crucial to thoroughly test the application to ensure that everything works correctly in all supported languages. Any bugs or issues should be identified and fixed as soon as possible. 6. **Continually update and maintain the translations over time**: As new features are added to the app, additional translatable strings may need to be extracted and translated into multiple languages. It is also important to periodically review and update existing translations to ensure that they remain accurate and up-to-date. By following these steps, developers can create applications that are accessible and easy to use for users all around the world.

Cloudflare has introduced IP Lists, a new feature that enables users to manage tens of thousands of IP addresses across all zones by grouping them in data structures. This enhances network-based security controls and allows for more complex logic implementation. IP Lists can be used with Firewall Rules engine to take action on incoming requests. The feature offers increased flexibility compared to previous tools like IP Access or Zone Lockdown rules, allowing users to create lists with any prefix length and combine them with other filter criteria. Additionally, IP Lists are stored at the account level, making it easier to reuse across multiple zones. Cloudflare plans to add more capabilities in the future, such as increasing quotas for paid plans, adding additional types of custom Lists, expiring List entries, and creating managed lists.

Every day, hundreds of thousands of new domains are registered globally, making it easy for attackers to create fraudulent domains for use in phishing campaigns. According to ICANN, nearly 5.45% of newly registered domains per day are malicious (including phishing, botnets, and malware). On July 16th, 2020, an email from a domain strikingly similar to the legitimate Bill & Melinda Gates Foundation's domain was sent to numerous recipients, seeking donations in Bitcoin. The attacker used typosquatting when creating the domain name and set up an SPF record for reliable delivery of their attack. This phish was sent just a day after Bill Gates’ Twitter account was hacked and used to tweet a message nearly identical to this email.

Masa Aoba has joined Cloudflare as the Head of Japan, with plans to expand the company's capabilities in the Japanese market and address technical pain points faced by businesses in the region. The first Japan office is set to open in Tokyo, aiming to grow the Cloudflare business and team there. Aoba highlights that Cloudflare helps solve challenges faced by businesses globally, including ensuring security, performance, and reliability online. He emphasizes his commitment to helping more customers in Japan and making a better internet part of their reality.

On July 18, 2020 at 2:22AM UTC, a configuration error in Cloudflare's backbone network caused an outage lasting 27 minutes, affecting Internet properties and Cloudflare services. The issue was localized to certain geographies due to the architecture of their backbone. Traffic dropped by about 50% across their network. The problem occurred when a configuration update on a router in Atlanta caused all traffic across the backbone to be sent there, overwhelming the Atlanta router and causing Cloudflare network locations connected to the backbone to fail. No attack or breach was involved. Cloudflare has made changes to prevent this from happening again and apologized for the disruption.

The text discusses how to use Cloudflare's Workers platform for server-side rendering of web pages at the network edge. It explains the history of web page development and how server-side rendering can improve SEO rankings, cacheability, and user experience. The author then demonstrates building a dynamic web page using Cloudflare Workers Sites, Wrangler, HTMLRewriter, and other tools from the broader Workers platform. They provide an example of Peer With Cloudflare (PWC), which is a user-maintained public database of networks, exchanges, facilities, and interconnection on the Internet. The PWC application uses the PeeringDB API to query live information on facilities and exchange points from multiple ASNs, compares the resulting networks, and lists shared exchanges and facilities. The author also explains how to use Handlebars templating language with HTMLRewriter for dynamic transformation of responses before returning them to users. Finally, they discuss caching at the edge using Cloudflare's Cache API.

In July 2019, John Graham-Cumming wrote about the opening of Cloudflare's office in Lisbon due to its attractive immigration policy, political stability, and high standard of living. A year later, the team has grown from 12 to 35 members with plans to reach around 80 by the end of 2020. The pandemic has affected travel but not the company's commitment to Lisbon as a key location for growth. Despite the pandemic, Cloudflare continues to invest in Lisbon and currently has 24 open roles across various departments. The team appreciates Lisbon's lifestyle, including its food, beaches, family-friendly environment, and fast internet access.

In 2020, despite the challenges posed by the COVID-19 pandemic, Cloudflare has continued to expand its global network, now reaching 206 cities in over 100 countries. This includes adding over 1Tbps in dedicated backbone capacity and completing more than 40 datacenter expansion projects. The company's Infrastructure department faced new challenges due to the pandemic, such as supply chain disruptions and grounded passenger flights affecting air freight capacity. However, with the help of existing logistics partners like Expeditors, Cloudflare was able to maintain its global infrastructure cargo flows. In addition to these accomplishments, six new cities have been added to Cloudflare's network this year, five of which represent new countries for the company: Vientiane (Laos), Tegucigalpa (Honduras), Johor Bahru (Malaysia), Monrovia (Liberia), Bandar Seri Begawan (Brunei), and Paramaribo (Suriname). These expansions not only bring the Internet closer to users in these regions but also increase Cloudflare's edge compute capabilities. The author of this text, Kevin Dompig, shares a personal connection with Suriname, where his family is from. He highlights how the addition of a data center in Suriname has improved internet access and reliability for users there. As more businesses and individuals rely on the Internet during these times, Cloudflare continues to grow its company by hiring remote workers and offering fully remote onboarding.

Cloudflare has introduced a new software-defined DDoS protection system called flowtrackd to enhance its Magic Transit service. This feature will be enabled by default at no additional cost on July 30, 2020 for existing customers. Flowtrackd significantly improves the ability to automatically detect and mitigate even the most complex TCP-based DDoS attacks. The system is designed to handle unidirectional TCP flows, which are common in Cloudflare's Magic Transit service. It works by determining whether to forward or drop each received TCP packet based on the state of its related connection. Flowtrackd complements Gatebot and dosd, providing a comprehensive multi-layer DDoS protection system.

On June 21, 2020, Cloudflare mitigated a highly volumetric DDoS attack that peaked at 754 million packets per second. The attack was part of an organized four-day campaign and targeted a single Cloudflare IP address mostly used for websites on the Free plan. No downtime or service degradation was reported during the attack, and no charges accrued to customers due to Cloudflare's unmetered mitigation guarantee. The attack was detected and handled automatically by Gatebot, their global DDoS detection and mitigation system without any manual intervention. During the four days, the attack utilized a combination of three attack vectors over the TCP protocol: SYN floods, ACK floods, and SYN-ACK floods. Despite the high packet rates, Cloudflare's edge continued serving its customers during the attack without impacting performance at all. The DDoS protection systems Gatebot and dosd were instrumental in detecting and mitigating the attack automatically.

Linux seccomp is a powerful security feature that allows applications to restrict their system call usage, thereby limiting potential attack vectors. By using seccomp, developers can create sandboxes for their applications without writing any additional code. This post explores the use of seccomp in practice and provides examples of how it can be used to protect against arbitrary code execution exploits. The Cloudflare sandbox toolkit is also introduced as a convenient way to enforce seccomp policies on both dynamically linked and statically linked applications.

On July 7, 2020, Cloudflare deployed a managed rule to protect its customers against a remote code execution vulnerability found in F5 BIG-IP's web-based Traffic Management User Interface (TMUI). The new rule automatically blocks any attempt to exploit the vulnerability. Initial testing showed that attackers began probing and attempting to exploit this vulnerability starting on July 3. F5 has provided detailed instructions for patching affected devices, detecting attempts to exploit the vulnerability, and adding custom mitigation. The most common probe URLs have been identified, all containing a critical pattern at their core. On July 3, there were approximately 1k probes, which increased to around 1m on July 6. Remote Code Execution (RCE) is a type of code injection that allows attackers to run arbitrary code on the target application, potentially leading to full system takeover. The vulnerability affects only the administration interface and not the underlying data plane provided by the application. To mitigate this issue, blocking all requests matching a specific regular expression in the URL can be effective. Cloudflare WAF users with their F5 BIG-IP TMUI interface proxied behind Cloudflare are automatically protected from this vulnerability using rule 100315.

Cloudflare has launched a new streaming platform called Cloudflare TV, inspired by 90s television shows that shared exciting developments in computing. The platform aims to provide continuous programming with easy guest participation, similar to joining a Zoom call. It is built using various technologies such as NGINX for ingesting RTMP feeds and creating HLS/DASH segments, Contentful for scheduling content, Brave for virtual rooms, and Workers for connecting these systems together. The platform currently features live streaming of Zoom calls and pre-recorded content, with plans to add closed-caption support, enable viewer participation, and expand to platforms like Apple TV and Roku in the future.

Cloudflare has made significant performance improvements to its Web Application Firewall (WAF) by transitioning from PCRE to RE2 and implementing memoization. The WAF now uses deterministic finite automaton instead of backtracking algorithms, resulting in a linear time execution with the size of input. Memoization was also introduced to cache the output of function calls for reuse in future calls, leading to significant savings. These changes have resulted in an increase of the cache hit percentage from 56% to 74%, and a sharp decrease of 40% in the average time the WAF takes to process and analyze an HTTP request at the Cloudflare edge. The company is currently porting its Lua WAF to use the same engine powering Firewall Rules, which uses a filter syntax inspired by Wireshark® for better performance and safety.