November 2018 Summaries

Cloudflare has launched the second iteration of its App Developer Fund, in partnership with Venrock, Pelion Venture Partners, and New Enterprise Associates. The fund aims to provide developers with resources to build apps using Cloudflare's network and serverless product Workers. This initiative seeks to foster a new economy that thrives on developers' serverless innovations. Apps developed should align with Cloudflare's mission of creating a safe, fast, and reliable internet. The company is looking for innovative applications in areas such as custom application security, data loss prevention, and bot mitigation. Developers from various backgrounds are encouraged to apply.

Cloudflare Workers, a JavaScript environment at the edge, can be used to send traffic logs to arbitrary locations. This post discusses an example implementation of how to achieve this using sub requests. The provided script assumes an Elastic stack setup and sends log data asynchronously to avoid adding latency to the main request. The logToElk function collects various fields from the HTTP request, including some specific to Cloudflare, and then uses a sub request to PUT the log line to the backend ELK stack. This method allows users to import Cloudflare traffic logs into their ELK stack in real time regardless of their plan. Additionally, Argo Tunnel and Access can be used to improve Kibana deployment security.

In 2018, Vlad Krasnov discussed how Cloudflare implemented TLS 1.3 on top of the Go TLS library and made a Go replica of nginx-ssl (go-ssl). The challenge was to make two different processes written in two different programming languages share the same TCP socket. They used Linux's SCM_RIGHTS feature, which allows passing file descriptors between applications using UNIX-domain sockets. This method enabled Cloudflare to implement TLS 1.3 without affecting customers or clients who did not enable it and allowed for quick iteration and frequent releases.

Cloudflare has introduced a new tool called L4Drop for efficient packet dropping as part of its distributed denial of service (DDoS) attack mitigations. The company's existing DDoS mitigation pipeline includes Gatebot, bpftools, iptables, and Floodgate. However, with the introduction of new Gen9 and ARM servers using different network interface cards (NIC), Floodgate is no longer compatible as it relies on a proprietary Solarflare technology to redirect traffic directly to userspace. To address this issue, eXpress Data Path (XDP) has been added to Linux, which uses an extended version of the classic BPF instruction set called eBPF. This enables high-speed packet dropping and allows arbitrary code to run for each packet received by a network card driver. L4Drop is based on XDP and has been implemented across all Cloudflare servers, protecting them against DDoS attacks. The company plans to continue improving its pipeline by supporting more simultaneous rules in L4Drop through multiple chained eBPF programs, increasing the efficiency of generated programs, and supporting new eBPF features.

On November 25, 2018, Andrew Fitch from Cloudflare hosted Real World Serverless events in multiple locations around the world. The Austin event featured two talks about Serverless technology by Victoria Bernard and Preston Pham from Cloudflare. Victoria discussed how Cloudflare Workers can improve a site's performance and add functionality through proxy workers, while Preston explained building a simple JavaScript routing library for serverless Worker scripts.

Cloudflare recently held its 9th annual company retreat in San Francisco, bringing together 760 employees from various global locations to bond, build bridges, have fun, and learn. The event focused on celebrating the company's positive culture and achievements over the past year, fostering an environment for employee development, and promoting a sense of camaraderie among team members. Activities included break-out sessions on topics such as financial literacy, mindfulness, whiteboarding, speed reading, mentoring, lock picking, and how to be a better writer. The retreat also provided opportunities for employees to engage with the executive team and participate in various recreational activities. Cloudflare is currently hiring, and interested candidates can visit their careers page to learn more about joining the company.

The text discusses an experiment conducted to observe the effects of dynamic RAM (DRAM) refresh cycles on software performance. It explains how DRAM stores data using tiny capacitors that gradually lose their charge, requiring periodic refreshes to maintain data integrity. The author uses a C program to measure memory load times and detect any noticeable delay caused by the refresh process. By applying Fast Fourier Transform (FFT) to the collected data, they identify a consistent frequency spike corresponding to the 64ms refresh interval. This experiment demonstrates that subtle hardware behaviors can be observed from simple userspace processes, which has implications for security vulnerabilities like Rowhammer and Meltdown/Spectre attacks.

Sevki Hasirci shares his experience of building Progressive Web Apps (PWAs) using Cloudflare's network and a single JavaScript bundle that runs both in the browser and on Cloudflare Workers without modification. The same code can be used for rendering pages, making it isomorphic. PWA features include offline functionality, background work, and notifications. This technique allows developers to build applications that run exactly the same on the server as they do in the browser, with no additional cost when online and still working offline if JavaScript is disabled or the browser doesn't support it. The code for this blog post is available on Cloudflare's GitHub.

This article discusses how to use Cloudflare Workers to modify HTML content in real-time, specifically focusing on embedding Google Fonts CSS into web pages. The author provides a detailed step-by-step guide on how to create and deploy a worker that intercepts incoming HTTP requests, modifies the HTML content, and then forwards the modified response back to the browser. The main benefits of this approach are improved performance (since the font CSS is cached at the edge), reduced reliance on JavaScript for page modifications, and better control over how third-party resources like Google Fonts are loaded into web pages. To achieve this, the author uses various features of Cloudflare Workers such as Streams API, TransformStream, TextEncoder/Decoder, and Cache API. The code provided demonstrates how to handle character encodings other than UTF-8, detect unsupported charsets, and modify HTML content while preserving its integrity. Overall, this article showcases the power and flexibility of Cloudflare Workers for enhancing web performance and security by offloading complex tasks from the server or client side to the edge network.

Erfi Anugrah shares his experience as a marketing intern at Cloudflare, highlighting the company's focus on privacy and security, diverse cross-functional teams, and innovative products like 1.1.1.1 DNS release. He emphasizes the importance of learning from different perspectives and being curious in order to grow within the organization. Anugrah also discusses his personal journey into web development and how it led him to apply for an internship at Cloudflare, a company whose mission aligns with his beliefs in advocating for free speech, privacy, and security on the internet.

Cloudflare has submitted comments on the National Telecommunications and Information Administration's (NTIA) proposed approach to advance consumer privacy. The company believes that internet infrastructure companies should be part of the conversation about the future of internet privacy, advocating for an accessible yet secure and private internet. NTIA is seeking feedback from stakeholders on a proposed approach to consumer data privacy, while another Department of Commerce agency, the National Institute of Standards and Technology (NIST), has begun a parallel process to develop an enterprise-level voluntary framework that businesses can use to mitigate privacy risks for consumers. Cloudflare supports federal legislation that serves the goals of advancing consumer privacy, protecting innovation, and enabling security research. The company also believes that accountability is essential to raising the bar on consumer privacy protections and suggests creating incentives to support privacy research.

Cloudflare recently held its annual retreat where over 750 team members from various offices gathered for three days of learning and bonding. One highlight was the smashing of a VPN piñata on stage with a baseball bat, symbolizing the company's successful transition to a faster and safer alternative for accessing internal applications. The latest feature added to Cloudflare Access now allows users to place applications requiring SSH connections behind it. This includes source control repositories. The move was made possible by leveraging other Cloudflare products like Argo Tunnel, which secures SSH connections, and Argo smart routing for accelerated performance. With this feature, teams can improve their daily workflow and enhance security while accessing internal applications.

Cloudflare has announced a public beta for its Cloudflare Apps with Workers, allowing developers to package all Workers capabilities and deliver new worker-powered experiences to Cloudflare customers. This integration brings together Cloudflare's serverless platform, enabling the next wave of innovation in the Cloudflare marketplace. Developers can now create more powerful apps using Workers code running on the Cloudflare network. The submission process for app approval and review will remain the same, with an additional security review for Worker-based solutions. Interested developers can apply to join the beta program.

Cloudflare has launched the Bandwidth Alliance with its founding partners to help reduce or waive data transfer charges for mutual customers. The alliance aims to make the internet faster, smarter, safer, and more cost-efficient by partnering with cloud providers and hosting companies. Companies joining the alliance include Data Space, DNSnetworks, Host1Plus, Vultr, and Wasabi. These partners offer a wide variety of choices for cloud services and products, helping customers take advantage of best-of-breed solutions without worrying about transfer fees. The alliance is open to other like-minded companies interested in joining.

On November 12, 2018, Google experienced a 74-minute outage due to a "route leak" caused by Nigerian ISP MainOne's accidental misconfiguration of their network. This resulted in unusual routing paths for Google and other networks, overwhelming the intermediary networks and causing services to become unreachable. Route leaks occur when traffic is rerouted outside regular and optimal paths. In this case, China Telecom CN2 did not filter MainOne's incorrect routing information, allowing the misconfiguration to propagate through their network and eventually reach Google. This incident highlights the trust involved in the underlying connections that make up the internet and demonstrates how a routing mistake can have widespread effects. Cloudflare is working with others to improve secure and stable routing practices, such as implementing RPKI validation and encouraging proper filtering of customer routes.

The Cloudflare Bandwidth Alliance has added over a dozen partners to its ecosystem, aiming to provide clients with modern, cloud-centric applications without the burden of egress fees. The alliance's partner landscape is diverse and continues to grow. To help customers choose the best fit for their needs, Cloudflare applies an approach that considers factors such as zero-egress storage options, costs, read and write requests, volume, size, API compatibility, and security. By understanding clients' specific requirements, Cloudflare can recommend suitable partners and design effective solutions.

Google has implemented HTTP signed exchanges, a crucial component of the emerging Web Packaging standard, which extends authenticity and integrity to content cached and served on behalf of a publisher. This solution allows AMP caches to serve content under its origin URL. The technical details involve using MICE (Merkle Integrity Content Encoding) for proof of integrity and CBOR (Concise Binary Object Representation) for serializing headers and payloads. Cloudflare Workers can automate the generation of HTTP signed exchanges, making it easier for publishers to distribute AMP content with their original URLs.

The blog post discusses various types of Layer 3/Layer 4 DDoS attacks that have occurred since the last write-up on this topic. It mentions a gigantic SYN flood in April, which was notable for its large size and global distribution. The attack seemed to be partially spoofed with random source IP addresses. Apart from the gigantic SYN floods, the author also talks about common SYN floods coming mainly from Asia, reaching 600-650 Gbps. Additionally, the blog post discusses the decline of SSDP and memcached amplification attacks, which were previously considered significant threats. The author notes that "multivector" amplifications have become more common recently, where attackers pool different amplification types to generate substantial traffic. The overall perspective is that while volumetric attacks still happen, they are less frequent and smaller in scale compared to a few years ago.

On November 11th, 2018, Cloudflare launched its mobile app for their public DNS resolver, 1.1.1.1. This service aims to provide users with a faster and more private internet experience by preventing their ISPs from selling browsing history data. The app is easy to use and free of cost. Since the launch of 1.1.1.1 on April 1st, many people have switched to it for improved performance and privacy. However, setting up this service was complex, particularly on mobile devices. The new mobile app aims to make it easier for everyone to access these benefits.

Cloudflare's cloud computing platform, called Workers, uses Isolates instead of containers or virtual machines for its Serverless and cloud computing services. This technology was developed by the Google Chrome team to power the Javascript engine in their browser. Isolates are lightweight contexts that group variables with the code allowed to mutate them, allowing a single process to run hundreds or thousands of Isolates seamlessly. Workers offer faster cold-start times and better memory management than traditional Serverless platforms like AWS Lambda, resulting in lower costs for users. However, an Isolate-based system can't run arbitrary compiled code, limiting its use to modern applications or legacy applications with latency-sensitive components initially.