August 2016 Summaries

On August 25, 2016, CloudFlare introduced an affordable option to increase the number of Page Rules for Free, Pro, and Business users. For $5 per month, users can add five additional Page Rules without upgrading their plan. Page Rules allow for customization of site speed and application of various features on specific parts of a website. They are also accessible via API for integration into build processes or synchronization across domains. Additionally, CloudFlare launched a tutorial site with videos to guide users in setting up Page Rules for content management systems like WordPress, Magento, and Drupal, as well as optimizing website speed, enhancing security, and reducing bandwidth costs.

The Consumer Product Safety Commission (CPSC) is indirectly responsible for facilitating some of the largest distributed denial-of-service (DDoS) attacks on the Internet due to a misconfigured Domain Name System Security Extensions (DNSSEC) implementation. DNSSEC is designed to prevent cache poisoning, but when implemented incorrectly it can be used by attackers to amplify their DDoS attacks. The CPSC's DNS zone file is currently over 4,000 bytes long and could be reduced to around 1,389 bytes without compromising any functionality or security. This would significantly reduce the maximum possible DNS reflection attack size using the CPSC's domain. The misconfigured DNSSEC implementation includes a number of mistakes that can be fixed by choosing better encryption algorithms, eliminating redundant records, and optimizing other aspects of their configuration. The CPSC could also benefit from implementing anti-DNS reflection protections similar to those offered by CloudFlare's DNS infrastructure. In order to prevent future misconfigurations like this one, it is important for organizations to ensure that they have proper training and resources in place when setting up and managing their DNS records. Additionally, the Internet Engineering Task Force (IETF) should consider implementing stricter standards or guidelines for configuring DNSSEC to help reduce the likelihood of similar issues arising in the future. Overall, while it may seem like a small issue, fixing misconfigured DNS zones can have a significant impact on reducing the potential harm caused by DDoS attacks and improving the overall security and stability of the Internet as a whole.

CloudFlare protects web sites from hackers, makes them faster, and ensures they work on various devices. The company aims to provide the most up-to-date internet experience for all websites regardless of their size without requiring users to install software or maintain hardware. As technology evolves rapidly, web site owners need protection from the latest security vulnerabilities and must supply a fast, rich experience to users. Hardware appliances are no longer suitable for this purpose as they get upgraded less frequently than mobile phones and are considered financial assets by some people.

In Cuba, where internet access is limited and expensive, many Cubans rely on an alternative network called "El Paquete Semanal" to access media content such as films, TV shows, music, mobile apps, magazines, and even web sites. This weekly service involves someone delivering a disk containing the latest downloads of various media for 2 CUC ($2) per week. The contents are diverse, including up-to-date antivirus updates, international and Spanish-language magazines, Android and iOS apps, TV shows, films, music, and even screenshots of web pages from popular websites. While this sneakernet approach is not a permanent solution, it provides Cubans with access to a wide range of media content that they would otherwise be unable to obtain due to the country's limited internet infrastructure.

Cloudflare, a global network provider, has recently updated its analysis of bandwidth costs in different parts of the world. The company's network now spans 86 cities across 45 countries and offers protection to over 4 million Internet properties. Bandwidth is the largest recurring cost for providing their service. In Europe and North America, transit costs are around 10 units per Mbps, with peering rates of about 60% and 40%, respectively. This results in an effective regional cost of 4 units in Europe and 6 units in North America. Africa has the highest transit prices globally at 140 units, but Cloudflare manages to peer around 90% of its traffic there, resulting in an effective cost of 14 units. In the Middle East, where Cloudflare can peer all its traffic, bandwidth costs are effectively zero. In Asia (excluding the Middle East), transit costs are seven times higher than in Europe or North America, but peering rates of around 60% result in an effective cost of 28 units. South Korea is a notable exception with transit costs increasing due to new government regulations. In South America, transit prices are very high at 170 units, and peering rates of about 60% result in an effective cost of 68 units. In Oceania (Australia and New Zealand), transit costs are also extremely high at 170 units, but peering rates of around 50% result in an effective cost of 85 units. Six expensive networks (HiNet, Korea Telecom, Optus, Telecom Argentina, Telefonica, Telstra) account for nearly half of Cloudflare's bandwidth costs despite representing less than 6% of the traffic. These providers have refused to discuss local peering relationships and are more than an order of magnitude more expensive than other bandwidth providers around the globe. To encourage these six networks to be on the right side of a free and open Internet, Cloudflare has moved its Free customers off these transit providers. The company is hopeful that it will soon be able to serve all its customers across all the networks they interconnect with.

In April 2016, support for HTTP/2 Server Push was announced via the HTTP Link header. Netjet middleware was developed to parse generated HTML and automatically add Link headers in Node.js applications. This technique can be implemented in other environments as well. However, using an HTML parser in the response stack increases page load latency. To mitigate this issue, netjet includes an adjustable LRU cache keyed by ETag headers. Netjet is compatible with any Node.js HTML framework that supports Express-like middleware and can be used without frameworks as well. Chrome's Developer Tools can verify Server Push usage while Firefox's Developer Tools do not directly expose it yet. Contributions to improve netjet or the documentation are appreciated. Netjet has been integrated with Ghost, a popular blogging platform, since version 0.8.0.

A network issue was debugged where connections going through CloudFlare would time out with a 522 HTTP error. The root cause of the problem was traced back to loopback congestion, which is not possible under normal circumstances due to the absence of buffering over loopback. Further investigation revealed that hundreds of CLOSE_WAIT sockets were present, indicating leaking sockets in the listening application. This issue caused some incoming connections to time out. The problem was resolved by addressing the socket leakage and ensuring proper handling of TCP states.

Vojtech Miksu shares insights into Cloudflare's RESTful API documentation process and introduces a set of tools used for generating the same. The company uses JSON Hyper-Schema to describe their API, which is then used to generate HTML documentation. They have an open source library called JSON Schema Docs Generator (JSDC) that generates complete HTML documentation from JSON Schema files and Handlebars.js templates. However, they found it hard to use for other teams due to its complicated configuration and slow performance. To address these issues, they created a toolchain based on JSDC and modern JavaScript libraries. They also introduced doca, a command-line tool that scaffolds the whole application for users. The tools are designed to be modular and extensible while still providing ready-to-go output with just a few commands.

In this blog post from August 2016, Gilberto Bertin discusses their love for BPF (BSD packet filter) bytecode and the utilities they use to generate BPF rules for production iptables. They also open source another component of bpftools: their p0f BPF compiler. The p0f tool is used to passively analyze and categorize arbitrary network traffic, extracting information about the operating system that sent a packet. The author explains how they use this on a daily basis at CloudFlare to categorize packets when they are a target of a SYN flood attack. They also describe the signature format used by p0f and how it can be used to distinguish different types of SYN packets, which helps in mitigating attacks. The author provides an example of how to compile p0f to BPF using their bpftools project and shares an example run where they block SYN packets generated by the hping3 tool. They conclude by encouraging others to work together on solving the DDoS problem for all, and inviting interested individuals to apply for open positions at CloudFlare's offices in London, San Francisco, Singapore, Champaign (IL), and Austin (TX).