March 2018 Summaries

Cloudflare has expanded its Europe network to 41 cities and its global network to 151 cities across 74 countries by completing provisioning for data centers in Luxembourg City and Chișinău. The company added 31 new cities in March alone, providing additional resilience and performance for a population of over one billion people. In total, Cloudflare now has six cohorts of data centers, with the most recent being completed in just 24 days. The expansion includes facilities in Beirut, Phnom Penh, Kathmandu, Istanbul, Reykjavík, Riyadh, Macau, Baghdad, Houston, Indianapolis, Montgomery, Pittsburgh, Sacramento, Mexico City, Tel Aviv, Durban, Port Louis, Cebu City, Edinburgh, Riga, Tallinn, Vilnius, Calgary, Saskatoon, Winnipeg, Jacksonville, Memphis, Tallahassee, Bogotá, Luxembourg and Chișinău. The company's aspiration for 2018 is to have data centers in countries where 95% of the world's Internet users live.

On March 30, 2018, Cloudflare opened its 149th data center in Bogotá, Colombia, making it the third highest capital city globally and the 29th city added by Cloudflare in March alone. This marks the company's 11th deployment in the Latin America and Caribbean Region, with the new facility located at a Tier III facility within the Bogota Free Trade Zone. The expansion aims to attract ICT investments and improve internet services for users in the region.

The author was tasked with extracting IP TTL values from TCP connections established by a userspace program to ensure that the connections are routed outside of their datacenter. They initially considered using raw sockets or the IP_RECVTTL socket option, but ultimately decided on attaching an eBPF filtering program to a socket. The BPF map abstraction allowed them to store data and share it with userspace code. By creating an "eBPF map" and writing an eBPF program, they were able to extract the TTL values of passing packets and analyze the recorded values from userspace. They also discussed the calling convention for eBPF programs, accessing packet payloads, a magical Layer 3 offset, and the return value of BPF programs. The author concluded by recommending keeping dependencies small when working with eBPF and noting that there is more to explore beyond network packets and socket programming.

Nitin Rao announced on March 29, 2018 that three new data centers have been launched in the Southern United States, specifically in Jacksonville (Florida), Memphis (Tennessee), and Tallahassee (Florida). These additions bring the total number of cities across Cloudflare's global network to 146, with 42 of them being in North America. The new data centers help reduce latency for millions of Internet properties using Cloudflare while expanding their capacity to handle attacks.

Cloudflare has expanded its network with new data centers in Calgary, Saskatoon, and Winnipeg, bringing its Canadian presence to six cities. The company is encouraging more ISPs to interconnect across the Prairies region. Michelle Zatlyn, co-founder of Cloudflare, shared her favorite things about the Canadian Prairies, including Calgary's proximity to Banff National Park and Saskatoon's emerging art scene. The expansion aims to make the internet faster, safer, and more reliable for users in the region.

On March 29, 2018, Cloudflare announced the launch of new data centers in Riga (Latvia), Tallinn (Estonia), and Vilnius (Lithuania). These additions represent the 140th, 141st, and 142nd cities across their global network, and the 37th, 38th, and 39th in Europe. The company aims to improve security and performance for over 7 million internet properties across 72 countries, including the Baltic states. Cloudflare will interconnect with local networks through multiple Internet exchanges, such as BALT-IX, LIXP, LITIX, TLLIX, RTIX, SMILE-LV, and potentially LIX-LV. The new data centers are part of the company's ongoing expansion in Europe and along the Baltic Sea region, which includes Denmark, Finland, Germany, Poland, Russia, and Sweden.

On March 29, 2018, Drupal announced an update to fix a critical remote code execution exploit (SA-CORE-2018-002/CVE-2018-7600). In response, Cloudflare has released a rule to block requests matching these exploit conditions for their Web Application Firewall (WAF). The rule can be found in the Cloudflare ruleset under the Drupal category with the rule ID of D0003. The official Drupal Advisory is available at https://www.drupal.org/sa-core-2018-002.

Blake Loring, a PhD student at Royal Holloway, University of London, discusses how web compression schemes can be exploited by attackers to extract secret information from encrypted messages using only the length of the response. This has led to attacks such as CRIME, BREACH, TIME and HEIST on TLS. To mitigate this issue, Loring worked at Cloudflare last summer on a project called cf-nocompress, which aims to develop a tool that automatically mitigates instances of these attacks without significantly impacting the effectiveness of compression. The solution involves selective compression, compressing only non-secret parts of a page, and using regular expressions to identify secrets within a response. A proof-of-concept implementation is available on GitHub.

Cloudflare recently joined the group of trusted Certificate Transparency (CT) log operators, contributing to the strengthening of this critical ecosystem. The company has developed a publicly accessible tool called Merkle Town that tracks the health and performance of all trusted logs. The dashboard provides statistics on certificate types, frequency of issuance, top CA issuers, and interdependencies between CAs and existing logs. As Chrome plans to require SSL certificates to be logged for trust, it is crucial that the CT ecosystem continues to expand with new log operators. Cloudflare's Merkle Town aims to help improve transparency in web encryption by providing valuable insights into the functioning of the CT ecosystem.

Cloudflare and Typeform are hosting three events in London and Barcelona over the next two weeks. The first event, "Building for a tech audience," will be held at the Cloudflare London office on March 27th, led by Marta Bondyra and David Sancho from Typeform. The second event, "The Rise of Edge Compute," will take place at the same location on March 28th with Kenton Varda, lead architect of Cloudflare Workers, as the speaker. The final event, "Cyber Security & Web Performance: The New Landscape," will be held at Typeform's office in Barcelona on April 3rd, featuring Jason Harmon, Chief Platform Officer at Typeform, and speakers from Cloudflare.

Cloudflare has announced support for Certificate Transparency (CT), an ambitious project aimed at improving online security by bringing accountability to the system that protects HTTPS. The company is introducing two new public-good services: Nimbus, a free and open certificate transparency log; and Merkle Town, a dashboard for exploring the certificate transparency ecosystem. Starting April 2018, Chrome will require Certificate Transparency for all newly issued, publicly trusted certificates.

On March 23, 2018, Marty Strong announced that a new data center in Edinburgh expanded their global network to 139 cities, with three of those being in the UK. The city is known for its rich culture and landmarks like Edinburgh Castle. Deploying in Edinburgh encourages ISPs to regionalize traffic away from London and Manchester. They are connected to IXScotland and actively seeking peering partners to build a stronger community.

On March 23, 2018, Cloudflare opened its newest Data Center in Cebu City, Philippines. The city is the second most populous metro area and oldest city in the country. Its central location provides excellent coverage to central and southern Philippines, complementing Cloudflare's existing location in Manila for serving the Manila Metro and northern Philippines. Cebu has a rich history of trade and business activity dating back hundreds of years, with influences from Spanish visitors to modern trade and shipping. The city is also known for its white sand beaches.

Cloudflare has introduced a free mobile SDK for Android and iOS developers to visualize and understand their app's network utilization. The tool helps identify slowdowns caused by balky or too frequent network calls, allowing developers to focus on optimizing the lowest-hanging fruit. It enables users to identify top N requests, slow requests, and requests most likely to fail, as well as inspect third party calls made through included libraries. The SDK is free with no cap on active users or metrics tracked, and integrates easily with existing engagement data tools like Mixpanel, Amplitude, and Heap. Privacy is a top priority, with the SDK not collecting any persistent identifiers and never selling collected data to third parties.

Cloudflare, a company that provides content delivery network services and DDoS mitigation, has expanded its network to 137 cities globally with new data centers in Durban (South Africa) and Port Louis (Mauritius). The expansion aims to improve the performance and security of over 7 million Internet properties across 69 countries. In March alone, Cloudflare launched new data centers in various cities including Beirut, Phnom Penh, Kathmandu, Istanbul, Reykjavík, Riyadh, Macau, Baghdad, Houston, Indianapolis, Montgomery, Pittsburgh, Sacramento, Mexico City and Tel Aviv. The company plans to add several more Africa data centers in the second half of 2018.

On March 15, 2018, a new data center was launched in Tel Aviv, Israel by Rita Kozlov, expanding their global network to cover 135 cities across 68 countries. Despite its small size and young age, Israel has a thriving tech scene with numerous renowned companies having offices in Haifa's Matam technology park. The country is also known for inventions like flash drives, Waze, and cherry tomatoes due to its excellent education system. Israeli natives are nicknamed "Sabras" after the cactus fruit, symbolizing their tough exterior but warm hospitality. In addition, more cities will be included in the network soon to further enhance internet speed.

In late spring 2018, IBM sought to expand its connections with Cloudflare, a company that provides DDoS protection, WAF, and Load Balancing. After evaluating various solutions for filling product gaps in DDoS, DNS, WAF, and load balancing, IBM selected Cloudflare due to its trusted presence around the globe, scalable architecture, rich set of well-documented APIs, competitive and predictable pricing model, and commitment to empowering enterprise customers. As a result, IBM launched "Cloud Internet Services," powered by Cloudflare, enabling IBM Cloud customers to easily purchase and configure mission-critical web performance and security solutions. This partnership also led to IBM becoming an authorized reseller of Cloudflare's entire suite of services.

On March 14, 2018, Cloudflare introduced its 134th data center in Mexico City, marking its entry into the Aztec nation. Prior to this, traffic to Mexico was served from other datacenters located mainly in McAllen, TX, Dallas, TX, and Los Angeles, CA. The new data center will provide faster access for over 21 million inhabitants of the Mexico City Metropolitan Areas to more than 7 million websites served by Cloudflare. This is the company's tenth datacenter in the Latin America Region.

On March 14, 2018, Cloudflare expanded its network in the United States by adding five new data centers in Houston (Texas), Indianapolis (Indiana), Montgomery (Alabama), Pittsburgh (Pennsylvania), and Sacramento (California). These locations represent states with a combined population of nearly 100 million people. The expansion increases performance, security, and reliability for Cloudflare's customers while supporting the next generation of 5G-enabled applications. With this addition, Cloudflare now has deployments in all ten most populous North American metropolitan areas.

On March 14, 2018, Cloudflare opened a new data center in Baghdad, Iraq, marking its 45th data center in Asia and 128th globally. This expansion serves the Middle East region and brings the company closer to every Internet user. The deployment also benefits ISPs like Earthlink, which was previously served from Cloudflare's Frankfurt data center. Iraq is home to over 40 million people and has a rich culinary history dating back thousands of years. Additionally, Baghdad is the first of eight new data centers joining the Cloudflare network that week.

Cloudflare has announced the general availability of its Cloudflare Workers service. The platform enables developers to run JavaScript code on Cloudflare's edge network using the standard Service Workers API and V8 engine. This allows for faster response times by running code closer to end users, reducing latency. Some potential uses include routing different types of requests to different servers, expanding HTML templates at the edge, applying access control to cached content, and implementing custom load balancing logic. The service is a paid add-on to Cloudflare's existing offerings.

On June 4, Cloudflare will be dropping support for TLS 1.0 and 1.1 on api.cloudflare.com. The dashboard will also be moved from www.cloudflare.com/a to dash.cloudflare.com and will require a browser that supports TLS 1.2 or higher. No changes will be made to customer traffic that is proxied through their network, though customers may decide to enforce a minimum version for their own traffic. Prior to June 4, API calls made with TLS 1.0 or 1.1 will have warning messages inserted into responses and dashboard users will see a banner encouraging them to upgrade their browser. The Transport Layer Security (TLS) is the protocol used on the web today to encrypt HTTPS connections. Standards bodies such as the Payment Cards Industry Security Standards Council (PCI SSC) and the National Institute of Standards and Technology (NIST) recommend disabling TLS 1.0 and 1.1. Almost all traffic seen by Cloudflare already uses TLS 1.2 or greater, so most users will not need to make any changes.

On March 9th, 2018, Cloudflare launched its 127th data center in Macau, making over 7 million internet-facing applications safer and faster. This marks the company's 44th data center in Asia. The new data center will help enhance Macau's unique entertainment scene, scenic spots like the Ruins of St Paul and Senado Square, as well as its foodie paradise. Additionally, visitors can experience the world's highest bungy jump from Macau Tower.

Cloudflare has opened its 126th data center in Riyadh, Saudi Arabia, partnering with Zain to provide stronger coverage and resilience for over 7 million internet properties across the Middle East region. This comes after recent launches in Reykjavík. At least four more Middle Eastern deployments are planned. The new data center will benefit from reduced latency for all users accessing applications via Cloudflare, while Saudi Arabia's young demographic and increased 4G LTE deployment suggest a growing internet user base.

Iceland, a small Northern European country known for its active volcanoes and geysers, is an ideal location for running power-intensive computing operations due to its abundance of green electricity and natural cooling. Companies are placing high-power machines in Iceland to take advantage of these unique conditions. Reykjavík, the capital city, has become Cloudflare's 125th global location. Iceland connects to the internet through three active fiber optic submarine cables: FARICE-1, DANICE, and Greenland Connect. Latency to the nearest Cloudflare locations in London and Copenhagen starts at 35 milliseconds, but by deploying in Reykjavík, it can be reduced to under 1 millisecond. Iceland's Network Information Centre runs RIX, the Reykjavík Internet Exchange, where Cloudflare is the only CDN network connected. With a population of just 330K, Iceland has become a hotspot for tourism, with over 2 million visitors last year. The country's commitment to women's rights and progressive policies make it an attractive destination for businesses like Cloudflare.

Cloudflare has evolved its analytics pipeline over the last year to provide HTTP traffic analytics to its customers. The old data pipeline was built in 2014 and faced challenges as the company grew, including a single point of failure (SPOF) for Postgres and Citus main, complex codebase, high maintenance cost, and many dependencies. To address these issues, Cloudflare replaced the old pipeline with a new one based on ClickHouse, an open-source column-oriented database management system. The new data pipeline is more scalable, fault-tolerant, and easier to operate, providing analytics for over 7 million customers' domains. Additionally, historical data was transferred from the old Citus cluster into ClickHouse during the switch. Looking forward, Cloudflare plans to develop products like Log Push and Logs SQL API to enhance its analytics offerings.

Dani Grant from Cloudflare is excited about participating in South by Southwest (SXSW) in Austin, Texas. They will be conducting three sessions at the event. The first session on March 9th will discuss how Wi-Fi networks work and what happens when attackers coax people into joining insecure networks. The second session on March 11th will focus on the impact of recent revocation of Net Neutrality rules on online video streaming. The third session on March 15th will be about hacking a Tesla Model S and the current state of automotive security. All sessions will take place at different venues within the Hilton Austin Downtown and JW Marriott.

Large DDoS attacks are possible on the modern internet due to IP spoofing, a technique where an attacker sends IP packets with a fake source IP address. These attacks can be divided into direct and amplification types. Direct attacks involve sending traffic directly against a victim IP, while amplification attacks involve sending traffic to vulnerable UDP servers that reflect and amplify it by sending large responses to the unknowing victim. Both types of attack require IP spoofing. Effective filtering preventing IP spoofing can only be done on the edge of the network - at the last-mile ISPs. The internet community has recognized this issue and written down BCP38, which outlines how source IP filtering can prevent IP spoofing.

Cloudflare has opened its newest data center in Istanbul, Turkey, marking its 124th global data center and 62nd country. This is also the company's 34th data center in Europe. The expansion comes as part of a plan to announce nearly one new data center per day throughout March 2018. Istanbul, with a population of over 16 million people, has a significant internet presence and high mobile broadband usage. Turkey is also experiencing rapid adoption of social media and e-commerce due to its young population.

In order to reduce the size of messages being sent through Apache Kafka, we decided to implement compression algorithms. We started with Snappy, which is a fast compression algorithm that provides good compression ratio for many types of data. However, it was not enough to keep up with our ingress and egress rates. We then switched to Zstandard, which is a modern compression algorithm promising high compression ratio and throughput, tunable in small increments. We found that using Zstandard level 6 provided the best compromise between compression ratio and CPU cost for our data. By implementing these compression algorithms, we were able to reduce the size of messages being sent through Apache Kafka by up to 4.5x compared to no compression at all. This allowed us to improve our network and storage utilization significantly.