December 2018 Summaries

Cloudflare has introduced an improved debugging experience for its serverless computing platform, Cloudflare Workers. The company added a Network panel to the inspector, allowing developers to view interactions between their worker and the origin, including request and response headers, time taken by the worker to reach and get data from the website, initiator of each request, and more. This feature is useful for debugging complex interactions between various requests on a page and subrequests coming from a worker. The implementation involved using the DevTools Protocol and reusing capnproto for JSON (de)serialisation support. Cloudflare encourages feedback on this new feature and suggestions for further improvements to enhance developer experience with Workers.

Cloudflare's Proudflare, an employee resource group for the company's LBGTQIA+ community, participated in Singapore's Pink Dot event on June 21st. The event celebrates pride and equality for the LBGTQIA+ community. Proudflare started in San Francisco in 2017 and expanded to Singapore in 2018. At the event, Proudflare members discussed various articles about LBGTQIA+ issues in tech in Asia Pacific, specifically in Singapore. The group also met other LBGTQIA+ individuals and shared stories and methods of efficacy. They are optimistic that Singapore will change its laws regarding non-cis people and activities. Proudflare encourages support through following their social media accounts and joining them at future events.

In December 2018, Cloudflare published a series of cryptography quizzes for users to solve during the festive season. The challenges included decoding a hex string from Wireshark, generating Time-Based One-Time Passwords (TOTP) without a shared secret, and finding hidden messages in RPKI data. The first five participants who solved all the puzzles correctly were promised some Cloudflare swag. The quiz was closed on January 1st, with many responses received, including 15 correct ones. Hints and solutions to the challenges are provided for those interested in learning more about cryptography and its applications.

The time to first byte (TTFB) of a website is the time from when the user starts browsing until the HTML code of the requested page begins to arrive. A slow TTFB has been a major issue for over ten years while running WebPageTest. According to recent test data, 20% of pages have a TTFB greater than 3 seconds and 80% start rendering after 5 seconds (10% taking more than 10 seconds). Additionally, 500 pages were larger than 15MB. A fast TTFB is crucial as it directly impacts other metrics; every millisecond improvement in TTFB translates to a millisecond saving in all other measures. However, a fast TTFB does not guarantee a quick overall experience but a slow one certainly does. The TTFB can be affected by several factors including redirections, DNS settings, connection configuration, SSL negotiation, and server response time for HTML. While most of these issues are easily fixed using services like Cloudflare, the server response time is often the hardest to resolve. The waterfall graph shows the server response time as a light blue bar in the first request and can be embarrassingly obvious when it's slow. In optimal conditions, the server response time should not exceed the orange socket connection bar that comes just before. The slowness of origin response could be due to various reasons such as server configuration, system load, backend databases, and communication systems with which it interacts, or even the application code itself. Identifying the root cause of performance issues usually involves development teams working with Application Performance Management tools to trace the slowest parts of the application and improve them. However, many site owners lack resources or knowledge for such investigations. In most cases, they had hired a developer to create their site or done it themselves on WordPress and hosted it with the cheapest hosting they could find. Hosting is generally designed to run as many sites as possible, not necessarily with maximum performance. Most of the HTML content isn't especially dynamic; it needs to change relatively quickly when the site is updated but for most parts of the web, the content remains static for months or years. There are special cases like when a user logs in (as an administrator or otherwise) where the content differs, but the majority of visits are from anonymous users. If HTML can be cached and served directly from the edge, then the performance improvement could be significant (up to 3 seconds faster in all metrics in this case). There are dozens of plugins for WordPress for caching origin content, but they require configuration (where to store pages) and their performance still largely depends on hosting. By moving content to edge-caching, we reduce complexity, eliminate the additional time to go back to the origin, and completely remove hosting performance from the equation. It can also significantly reduce the load on hosting systems by offloading all anonymous traffic. Cloudflare supports static HTML edge caching, and commercial and enterprise customers can allow users with active sessions to skip cache by enabling "avoid cookies storing". This works in tandem with the Cloudflare plugin for WordPress, so the cache can be cleared when content is updated. There are also other plugins that integrate with various Content Delivery Networks but they all need configuration with API keys and implementations specific to each CDN. To make edge-caching widely adopted, we need a way to automatically (or as close to automatic as possible) cache HTML. For this purpose, we need a communication pathway between an origin (like a WordPress site) and an edge storage (like Cloudflare's edge nodes) to manage a remote cache that can be explicitly purged. The origin should be able to: Detect when it is facing an edge-compatible storage. Specify what content should be cached and for which visitors (for example, visits without login cookies). Purge stored content when it has changed (globally across all edges). Instead of requiring the origin to communicate with an API to purge changes and manual configuration to determine what to cache and when, we can do everything with HTTP headers on requests going back and forth between the edge and the origin: 1. An HTTP header is added to requests going from the edge to the origin to announce that there's an edge storage and its capabilities: x-HTML-Edge-Cache: supports=cache|purgeall|bypass-cookies 2. When the origin responds with a cacheable HTML page, it adds an HTTP header in the response to indicate that it should be cached and the rules for when the stored version should not be used (to allow bypassing caching of logged-in users): x-HTML-Edge-Cache: cache,bypass-cookies In this case, the HTML code will be cached but requests with cookies starting with "wordpress" or "wp-" in their name will avoid caching and go to the origin. 3. When a request modifies the site's content (updates a post, changes a theme, adds a comment), the origin adds an HTTP response header indicating that the cache should be purged: x-HTML-Edge-Cache: purgeall The only tricky part of managing this is that the purge has to remove the cache globally. The Worker's caches are local to each edge and do not provide a global interface for making operations. One way to achieve this is by using Cloudflare's API to purge the global cache, but it's a bit heavy-handed (it purges everything from cache, including scripts and images) and requires some configuration. If you know exactly which URLs will change when content is updated, doing a targeted purge in the API only of those URLs would probably be the best solution. Using the new KV store for Workers, we can purge the cache in a different way. The worker script uses a caching versioning scheme where each URL gets a version number added to it (for example, http://www.example.com/?cf_edge_cache_ver=32). The modified URL is only used locally by the Worker as a key for stored responses and the current version number is saved in KV, which is a global store. When the cache is purged, the version number is incremented, changing the URL of all resources. Older entries will naturally fall out of cache as they won't be accessed. It requires a small adjustment to set up KV for Worker, but hopefully in the future it can be automatic. I believe there's great value in standardizing a way for edge storage and origin to communicate about caching dynamic content. I would encourage content management systems to build direct support on platforms and provide a standard interface that could be used with different providers (even for local edge-caching on load balancers or other reverse proxies). After doing some more testing with different types of sites, I'm considering bringing the concept to the HTTP Working Group at IETF to see if we can create an official standard for control headers (using different names). If you have any feedback about how it should work or what features it should expose, I would love to hear from you (like purging specific URLs, varying content for mobile/desktop or by region, expanding it to cover all types of content, etc.).

The Time to First Byte (TTFB) of a website is the time from when a user starts navigating until the first HTML code for the requested page arrives. Slow TTFBs can significantly impact website performance, and fixing them can improve other metrics as well. Many factors can affect TTFB, including server response time, which is often the most significant and challenging issue to resolve. One solution is edge-caching of static HTML content, which can be implemented using plugins like Cloudflare's Business or Enterprise plans. Automating this process requires a method for communication between an origin (such as a WordPress website) and an edge cache (like Cloudflare's Edge nodes), allowing for remote cache management and explicit purging of cached content when changes occur. This can be achieved using HTTP headers to manage the caching process, making it easier for developers to implement this functionality without requiring API integration or manual configuration.

The Time to First Byte (TTFB) is a crucial metric in determining the speed of a website. A slow TTFB can significantly impact other metrics and result in a poor user experience. Various factors contribute to a slow TTFB, including redirects, DNS, connection setup, SSL negotiation, and server response time for HTML. One solution to improve TTFB is edge caching of HTML, which involves pushing the content cache further out to the edge, reducing complexity, eliminating additional time to get back to the origin, and removing hosting performance from the equation. Cloudflare supports caching static HTML, and business and enterprise customers can enable logged-in users to skip the cache by enabling "bypass cache on cookies." Automating this process using HTTP headers for communication between an origin (like a WordPress site) and an edge cache (like Cloudflare's edge nodes) is being explored.

In December 2018, a team from Cloudflare conducted an extensive review of 42 varieties of mince pies available in the South East of England. The methodology involved rating each pie on various characteristics such as innovation, booziness, pastry to filling ratio, pastry, and overall satisfaction. Data analysis was performed using Pearson Correlation Coefficient (PCC), which revealed that satisfaction with a pie's mince filling is the most likely criteria for overall pie satisfaction. The top three pies were M&S Bakery Mince Pies, Gail’s Bakery Mince Pies, and Iceland Luxury Mince Pies.

The Cloudflare Athenian Project was launched one year ago to provide free Enterprise-level service to election and voter registration websites run by state and local governments in the US. Since then, it has helped over 100 entities in 24 states protect their websites from various malicious attacks aimed at undermining the integrity of elections. On November 6th, the project team mobilized to help on-board over 30 new county-level websites and managed unpredictably large amounts of legitimate traffic during the US midterm elections. The aggregated election day data showed a significant increase in engagement with nearly three times the number of requests compared to September or any other month preceding it. No evidence of coordinated attacks across the election websites was found, but various attacks were stopped by rules within Cloudflare's Web Application Firewall (WAF). Looking forward to 2019, the project aims to continue improving its reach and encouraging SSL adoption among website administrators.

Over the past few months, a pilot project was conducted with Facebook to test the feasibility of securing the connection between 1.1.1.1 and Facebook's authoritative name servers using TLS encryption. The results showed that while initial connection adds some latency, it is offset by many queries. The DNS latency between 1.1.1.1 and Facebook's authoritative name servers was found to be comparable with average UDP connections. For more detailed information on the pilot project, visit Code, Facebook's Engineering blog.

Cloudflare has introduced a new feature allowing customers to change the ordering of their firewall rules. This enhancement was made in response to user feedback regarding the default precedence of rules and the need for more control over rule execution order. Customers can now choose between two methods - Priority Numbering, which is useful for managing large numbers of rules or using API/Terraform configurations, and Drag and Drop Ordering, which has a 200-rule limit. The Firewall Rules documentation provides further details on how these features work.

Cloudflare has announced the addition of ten new data centers across various countries including the United States, Bahrain, Russia, Vietnam, Pakistan and France (Réunion). These new data centers will help improve performance and security for over 12 million domains that collectively represent about half a billion internet users. The expansion brings Cloudflare's global network to span 165 cities with 46 new cities added this year alone.

A recent blog post by Junade Ali discusses the insecurity of Partial Password Validation, a practice used by many websites including banks and services that contain sensitive data. This method involves prompting users to provide three random characters from their passwords to validate account ownership. However, this approach can lead to weak password management and increased vulnerability to credential stuffing attacks. Ali conducted simulations using a database of 488,129 breached passwords and found that the presence of only three characters of a password is sufficient to let attackers breach a significant proportion of such accounts. The post argues that Partial Password Validation does not effectively protect against keyloggers and instead recommends using Two Factor Authentication or Multi Factor Authentication for enhanced security.

Dollar Shave Club's Lead Software Engineer, Hank Jacobs, introduced Cloudworker, a local Cloudflare Worker runtime that enables developers to run Cloudflare Worker scripts locally or anywhere they can run a Docker image. The tool aims for compatibility with Cloudflare Workers and supports WebAssembly execution and an in-memory version of the beta Workers KV feature. Since its release, Cloudworker has become an integral part of Dollar Shave Club's development workflow for their edge router and QA environments.

In 2018, Cloudflare Workers moved from beta to general availability and expanded its footprint to 155 locations. The company held a Real World Serverless event series in various cities, discussing serverless application development insights and new services like Cloudflare Workers KV. Three talks were given at the Singapore event: fundamentals of serverless technology by Tim Obezuk, twelve factors of serverless application development by Stanley Tan, and achieving no ops at scale with network-based serverless by Remy Guercio.

Cloudflare has introduced a new feature for its Apps that allows apps to automatically set up and manage configurable DNS records on more than 12 million registered domains on the Cloudflare network. This aims to alleviate the common difficulties people face with managing DNS records, especially those who are not technically proficient. The Pointless DNS app is a demonstration of this feature, which automates the management of a TXT record on any root or subdomain. Developers can now build robust and powerful apps that automate DNS record management, saving countless developer hours spent manually configuring records to integrate with solution providers.

In this guest post by Ben Ross, the Founder and CTO of POWr.io, he discusses the importance of DRY (Don't Repeat Yourself) in software engineering and how it applies to his company's culture. By automating repetitive tasks, employees can become Scaled Employees, making a multifold impact compared to an average employee in their field. Ross shares an example of integrating 12 POWr apps into Cloudflare using a single integration template and a few lines of code, saving time and resources. He also introduces POWr Apps as customizable tools for websites, emphasizing the value of using existing tools to maximize impact instead of reinventing the wheel.

Cloudflare has announced early access for Traffic Acceleration with its Cloudflare Mobile SDK. The feature uses novel transport algorithms to accelerate apps beyond the performance they would see with TCP. Enabling Acceleration through the SDK reduces latency, drives down network timeouts, and improves app user experiences. The Mobile SDK allows developers to measure and improve the speed of their app's network interactions.

In this project, we aimed to improve LuaJIT's performance consistency by implementing separate counters for each loop and function in a program. We also started work on implementing a new garbage collector (GC) for LuaJIT based on Mike Pall's suggestion. We found that the original version of LuaJIT is highly non-deterministic, which can be confusing for users and create problems when estimating server provisioning. By implementing separate counters, we were able to make performance more consistent overall, particularly for larger programs. However, this change also delayed some loops from being traced, which disadvantaged small deterministic benchmarks where loops are highly stable. We also discovered that the odd performance in some programs was related to LuaJIT's Garbage Collector (GC). When we moved from the 32-bit to 64-bit GC, the problem seemed to go away. This led us to reevaluate LuaJIT's GC and start work on implementing a new one based partly on Tom's previous work and also that of Peter Cawley. In conclusion, while we did not achieve everything we wanted to in 12 months, we made significant progress towards improving LuaJIT's performance consistency and developing a new GC for it.

The text discusses the implementation of an OAuth 2.0 Authentication server using Cloudflare Workers to simplify the process, reduce latency, and segregate service logic from the authentication layer. It outlines the steps of the OAuth workflow and provides a detailed walkthrough of the implementation, including setup, accepting page after callback, redirecting back to consumer, code to token exchange, giving the token to the consumer, and validating tokens for accessing resources. The author also mentions that a follow-up blog post will cover an OAuth consumer implementation.

Black Friday and Cyber Monday are significant events for online retailers, as evidenced by the substantial increase in page views and checkout interactions observed across various regions during these shopping holidays. Mobile browsing dominates overall, but desktop devices tend to lead to more conversions. Retailers should invest in optimizing their mobile and desktop ecommerce experiences to capitalize on this increased traffic.

On December 11, 2018, Cloudflare introduced an improved customer dashboard with enhanced features driven by customer feedback. The new zone overview page provides better visibility of key analytics and easy access to common settings changes. This redesigned page is responsive across various devices and screen widths. The release also marks the beginning of improvements in customer experience for 2019, as it was developed using Cloudflare's new prototyping framework. Users are encouraged to provide feedback on the new overview page.

In September 2018, the European Commission proposed a legislative measure to address the removal of terrorist content online. The proposal has been met with concerns from various internet companies including Cloudflare, due to its potential legal implications, practical application and possible unintended consequences. Some of these concerns include the broad definition of "terrorist content" which could lead to the removal of legitimate content, the one-hour timeframe for content removal, and the privatization of law enforcement. The proposal also does not account for the complexity and range of information society services having a storage component. Cloudflare suggests that if this proposal moves forward due to political pressure, additional due process should be required and the proposal should be significantly narrowed.

On December 4th, 2018 at 1:00 PM, Matthew Williams discussed the importance of website performance and how quickly a blog post loads can impact user experience. Most users prefer webpages that load under 5 seconds and are more likely to leave if it takes too long. Cloudflare's mission is to improve internet performance by sharing educational content on website speed optimization. They recently launched a Performance Learning Center, which covers topics such as SEO benefits of faster websites, network latency causes, and mobile-friendliness. The learning center aims to help anyone with a web property understand the complex topic of website performance.