March 2013 Summaries

The New York Times reported on a massive DDoS attack against Spamhaus, which was mitigated by CloudFlare. This attack is considered the largest known attack ever on the Internet. Initially, it sent 85Gbps of traffic, but later increased to more than 100Gbps. The attackers changed their tactics and targeted network providers used by CloudFlare for bandwidth. They also attacked the infrastructure of several major internet exchanges (IXs), causing congestion across multiple Tier 1 networks in Europe. This incident exposed vulnerabilities in the architecture of some IXs, prompting suggestions to improve security measures. The attack highlighted the problem of open DNS recursors and their potential threat to the stability of the Internet.

The text describes a Parallels Summit event where experts in the hosting and service provider industry gathered to share insights on current trends and news. CloudFlare co-founder Michelle Zatlyn conducted live interviews with 15 leading experts, including Dan Havens (VP of Sales at Parallels), Sam Renkema (CEO of Spam Experts), Marco Howen (CEO of LuxCloud), Terry Gerstner (VP of Marketing at MobeeArt), and Michele Nelyon (CEO of Blacknight). The summit included activities such as limo rides for attendees and "Conversations with CloudFlare" hosted at the booth. Videos from these interviews are being released in parts, with highlights and clips from five conversations provided in this text.

CloudFlare has introduced an update to its Page Rules feature, making it easier for users to manage their page-by-page settings. Previously, reordering Page Rules was a cumbersome process that required deleting and re-adding them in the desired order. Now, users can simply click on the icon next to each rule and drag it up or down to change its position. This update is available for all customers, with the number of Page Rules allowed depending on their plan type.

CloudFlare, a company that deals with large-scale Distributed Denial of Service (DDoS) attacks daily, shares the story of how they mitigated a 75Gbps attack against Spamhaus, an anti-spam organization. The attack was primarily carried out through DNS reflection, where the attacker sends requests for a large DNS zone file with the source IP address spoofed to be the intended victim to numerous open DNS resolvers. These resolvers then respond to the request, sending the large DNS zone answer to the intended victim. The attackers' requests themselves are only a fraction of the size of the responses, effectively amplifying their attack. To mitigate such an attack, CloudFlare uses Anycast, which announces the same IP address from every one of its worldwide data centers. This dilutes the attack by spreading it across facilities and ensures no single point on the network acts as a bottleneck. Once diluted, the attack becomes relatively easy to stop at each data center. The article also highlights the need for service providers to take serious efforts in closing open DNS resolvers, which are becoming the scourge of the Internet and leading to larger DDoS attacks.

Words Pictures Ideas (WPI), a Cloudflare customer, commemorated the 75th anniversary of San Francisco's Bay Bridge with "The Bay Lights," the world's largest LED light sculpture. As anticipation for the event grew, Brian VanderZenden, Lead Developer at WPI, reached out to Cloudflare to ensure TheBayLights.org could handle increased traffic. By implementing Cloudflare's "Cache Everything" feature, the site saw a significant increase in traffic with no issues and reduced system resource utilization.

On March 7, 2013, John Graham-Cumming announced that CloudFlare is sponsoring and helping to organize a new Go London User Group. The first meeting will take place on March 27 at Makers Academy, with CloudFlare providing food and drink. Speakers will be announced closer to the date. Sign up for the event as there are limited spaces available. Suggestions for speakers and talks of interest can be made in the comments section. Additionally, CloudFlare is actively hiring Go (and other) programmers in both London and San Francisco.

CloudFlare experienced an hour-long outage last weekend due to a flaw in their architecture which they are working to eliminate. The company has designed its systems with the assumption that failure is inevitable and thus must be planned for at every level. They use Anycast routing, both on the wide area network (WAN) and local area network (LAN), to handle failures gracefully. In case of a process crash, server crash, switch crash, or router crash, traffic fails over to the next closest data center. The company is constantly working to improve its fault tolerance and is currently hiring for positions related to building a robust network.

On March 3rd, 2013 at 09:47 UTC, CloudFlare experienced a system-wide failure of its edge routers, causing an outage that affected all of its services including DNS and web proxies. The cause was traced to a bad rule in the Flowspec protocol, which led to routers consuming all their RAM and crashing. The issue was resolved within 30 minutes by removing the bad rule and performing hard reboots on unresponsive routers. CloudFlare is working with Juniper to investigate the issue further and improve its network protocol controls.