October 2015 Summaries

Filippo Valsorda, a developer at CloudFlare, recently experienced an issue with their Go DNS server, RRDNS. The problem occurred when deploying a new module for the authoritative server, which adds IP addresses to DNS answers. After deployment, RRDNS stopped answering queries for beta domains served by the new code. The issue was caused by forgetting to call f.IPTable.RUnlock() in the getCFAddr function, leading to a deferred deadlock when the table reload function ran. Although no customers were affected due to testing on off-production machines and limited use of the new code, this incident highlights the need for better tooling to detect such issues. CloudFlare is currently hiring developers for their DNS server team in London, San Francisco, and Singapore.

Ben Fathi shares his experience of joining Cloudflare, an internet security and optimization company, despite having a background in operating systems. He was impressed by the company's distributed system architecture and its ability to handle large-scale operations across multiple data centers worldwide. The company has already processed more than 5% of all web requests and continues to grow with significant funding rounds and collaborations with major tech companies. Fathi sees an opportunity to build a fully distributed internet operating system at the edge of the network, improving end user experience and making the internet more secure and reliable. Cloudflare is currently hiring for positions in San Francisco, London, Singapore, and Washington DC.

The benchmarking results show that on average, Brotli at the maximal quality setting produces 1.19X smaller results than zlib at the maximal quality. However, for files smaller than 1KB the result is 1.38X smaller on average, a very impressive improvement, that can probably be attributed to the use of static dictionary. Compression speed-wise, Brotli at quality level 4 is slightly faster than zlib level 8 while having comparable compression ratio. However, most files are smaller than 64KB, and if we look only at those files then Brotli 4 is actually 1.48X slower than zlib level 8! For on-the-fly compression, the most important question is how much time to invest in compression to make data transfer faster. It seems that for files greater than 64KB, Brotli at quality setting 5 can speed up slow connections.

On October 20, 2015, Cloudflare announced that its private beta for DNSSEC (Domain Name System Security Extensions) is now open to the public. DNSSEC aims to protect against on-path attacker attacks by adding cryptographic hashes and signatures for authenticating DNS records. The beta is available to all websites using Cloudflare for their DNS services, with interested parties encouraged to email [email protected] for access.

A new brute force attack method for WordPress instances has been identified by Sucuri, allowing attackers to try a large number of username and password login combinations in a single HTTP request. This latest technique is harder to detect since it doesn't necessarily flood the server with requests. The vulnerability can be exploited using XML-RPC, which uses XML encoding over HTTP to provide a remote procedure call protocol commonly used for APIs and automated tasks in WordPress instances. CloudFlare paid customers have the option to enable a Web Application Firewall ruleset to stop this new attack method.

The text discusses how the Linux kernel network stack has performance limitations, especially when dealing with large packet floods. To address this issue, the author's company decided to contribute to the Netmap project. They introduced a "single RX queue mode" that allows selected network flows to be offloaded to a user space application for high-speed filtering while leaving other packets to be handled by the kernel as usual. The patch is awaiting code review and can be found on GitHub. The author also provides an example of how to run this with an Intel IXGBE NIC, demonstrating that it can handle about 5.8 million packets per second.

At CloudFlare, efforts are made to accelerate customer websites by reducing image sizes using open source tools like jpegtran, gifsicle and pngcrush. However, these tools can be computationally expensive. Recently, it was noticed that ten times more time is spent "polishing" jpeg images compared to PNGs. The performance of pngcrush was improved by using a supercharged version of zlib. Next, the focus shifted to optimizing jpegtran (part of libjpeg distribution). Using profiling tools like Linux perf utility, it was found that 83.5% of time is spent in just two functions - encode_mcu_AC_refine and encode_mcu_AC_first. Both these functions deal with Huffman coding of the jpeg image. Optimizations were made using SIMD (Single Instruction Multiple Data) instructions, resulting in a 2.25X speedup for the test image. Further optimizations are planned to improve performance further.

On October 6th, 2015, Cloudflare expanded its network with a new point of presence in Manchester, UK, bringing the total to 63 points across 33 countries. This expansion also made the UK part of an exclusive group of countries with more than one Cloudflare data center. The Manchester and London data centers provide redundancy and content localization within the UK for all customers. Manchester has a rich history in technology, being home to significant inventions such as the first modern computer. This expansion also made Cloudflare a member of IX Manchester, a peering exchange serving Northern England. For users, this means faster, safer, and more reliable web experiences with no additional effort required.