December 2016 Summaries

On December 30, 2016, Cloudflare opened its newest data center in Panama City, becoming the company's 102nd global data center and expanding their network to span across 50 countries. This expansion will improve internet access speeds for over 5 million internet properties in Panama. The country is experiencing significant economic growth with a high mobile device penetration rate but only half of its population has access to the internet, leaving room for future growth. Cloudflare plans to continue expanding their network in 2017 across multiple cities and continents to contribute to a safer and faster internet experience worldwide.

In 2016, internet connectivity was disrupted during elections in Gabon and Gambia, while DDoS attacks continued throughout the year affecting websites big and small. The Internet of Things became a tool for cyberattacks with botnets attacking DNS service provider Dyn. However, there were also positive developments such as increased adoption of IPv6, TLS 1.3 going live to improve internet security, and widespread support for HTTP/2 including Server Push. In the year ahead, Cloudflare predicts that 1Tbps DDoS attacks will become the baseline for massive attacks, IPv6 will become the norm in mobile networks, a SHA-1 collision will be announced, Layer 7 attacks will rise and mobile traffic will account for 60% of all internet traffic.

This post demonstrates how to test RESTful APIs in an automated fashion using PHP by building a testing framework through creative use of two packages - Guzzle and PHPUnit. The resulting tests will be something you can run outside of your API as part of your deployment or CI (Continuous Integration) process. The author explains how to set up the environment, configure PHPUnit to use Guzzle, write tests against an endpoint at httpbin.org, and provides a conclusion on the overall code.

This article discusses how to tune and harden a Go server for Internet exposure. It covers topics such as configuring crypto/tls for better security and performance, setting timeouts in net/http, enabling HTTP/2, using TCP Keep-Alives, managing ServeMux, logging errors, and monitoring metrics. The author emphasizes the importance of upgrading to Go 1.8 for improved features and stability.

Cloudflare's automatic image optimization feature, Polish, now supports Google's WebP format for customers on paid plans. WebP is a modern image format that can significantly compress images compared to older formats like JPEG and PNG. The new feature in Polish allows it to swap out an image with an equivalent compressed using WebP when the browser supports it. Customers on Pro, Business, and Enterprise plans can enable automatic creation of WebP images by checking the WebP box in the Polish settings for a zone. Studies show that WebP conversion achieves significant size improvements not only for JPEG images but also for PNG and GIF images, resulting in lower bandwidth and faster website delivery.

The text discusses how Salt is used by Cloudflare for managing their global fleet of machines, including configurations and network automation tasks. They decided to manage their own DNS records from Salt as well. After internal testing and finding bugs in their API, they are announcing the public availability of the Cloudflare Salt module. The module allows users to configure their DNS records via Salt, with a single source of truth, peer-reviewed, audited, and versioned changes. They plan to "saltify" other settings like WAF, caching, page rules, and more in the future.

In December 2016, Matthew Prince discussed how Cloudflare has grown from handling 3.5 billion pageviews a month in 2011 to over 1.3 trillion per month today. The company launched the Cloudflare Apps platform six years ago and has powered more than a million app installations onto customers' websites. However, with just 21 apps available, they knew there was room for improvement. In June 2016, Cloudflare met Eager, an app store focused on enabling non-technical website owners to install powerful tools through a slick interface. Impressed by their platform and the team's experience in building similar tools, Cloudflare decided not only to partner with them but also to acquire Eager. Eager co-founders Zack Bloom and Adam Schwartz joined Cloudflare along with their entire team, forming the core of the new Cloudflare App development team. They are currently integrating Eager's app store into Cloudflare, replacing the current marketplace with a next-generation app platform. A developer preview of the new app store will begin for app creators in January 2017, and the app platform itself will be released to customers in April 2017. Eager's current platform will sunset starting today, but apps previously installed on customer sites will continue to function indefinitely.

Zack Bloom and his co-founder Adam initially developed an app store for websites that allowed website owners to find and install various helpful tools. Their goal was to make it easier for developers to create and distribute their apps without relying on marketing or sales teams. After discussing integration with Cloudflare, they decided to join the company due to its potential for rapid growth and scale. They plan to release their platform as Cloudflare Apps in April of next year, allowing developers to reach millions of websites. The new platform aims to make it easier for website owners to access helpful tools and for developers to create and distribute their apps.

On December 12, 2016, Cloudflare introduced a new feature for its Business plan users called "Bypass Cache on Cookie." This feature allows websites to cache the HTML of anonymous page visits without affecting dynamic content. By caching anonymous page views, Cloudflare helps ensure that origin web servers don't waste time constantly regenerating pages which change rarely, reducing load times and server load for users. The "Bypass Cache on Cookie" setting can be customized with wildcard operators and OR pipe operators, and the cache TTL can also be set using the "Edge Cache TTL" option in Page Rule settings.

The blog post describes a debugging adventure on Cloudflare's Mesos-based cluster, which is primarily used to process log file information and detect attacks. Engineers encountered an issue where internal DNS queries were returning "no such host" errors for existing domains. Through extensive testing and analysis, it was discovered that the problem stemmed from packet loss during DNS resolution attempts. The solution involved increasing the retries option in the resolv.conf file to better handle transient network issues and improve the reliability of DNS resolution.

The recent surge in DDoS attacks has sparked debates within the DNS community about how to strengthen DNS against future attacks. One such proposal involves using an obscure feature of the core DNS protocol, known as DNS TLD glue records with custom TTL values. While this feature is not currently practical for DDoS mitigation due to operator apathy, it could be made more useful with a small tweak. The author argues that adjusting DNS glue TTLs would reduce the recovery time for DNS servers under attack and improve overall resilience against DDoS attacks.

In November 2016, a new pattern of DDoS attacks was observed by John Graham-Cumming and his team at Cloudflare. Unlike previous years where such attacks were mostly concentrated during weekends, this time the attackers seemed to be working regular hours, launching attacks daily from 6:30 PM UTC until early morning hours. The attacks peaked at 400 Gbps and lasted for about 8.5 hours each day. Interestingly, these attacks were not linked to the Mirai botnet but used different attack software targeting TCP protocols. Despite the intensity of these attacks, Cloudflare's systems successfully mitigated them without any impact on customers. The company is currently hiring for roles related to developing such robust systems.

On December 1st, the West African country of The Gambia experienced an internet blackout just before its presidential election. This is not unprecedented as similar actions have been taken by other governments during elections to prevent monitoring and reporting of voting incidents or rumors from spreading. However, blocking internet access also prevents citizen reporting of election-related violence. In Gambia's case, the incumbent President Yahya Jammeh conceded defeat after the election results were announced, surprising many people including himself. The reasons for the internet blackout are not clear but it is possible that it may have infuriated voters and increased the vote against the president. Internet access has become a significant part of daily life for many Gambians, and blocking it can have unintended consequences. Cloudflare continues to fight for free speech online through encryption deployment and programs like Project Galileo. The company participates in events such as the UN's Internet Governance Forum (IGF) to promote policies that will make the internet more reliable and open.