June 2012 Summaries

A huge storm in 2012 caused power outages at Amazon Web Services (AWS) data center in Virginia, leading to service interruptions for many companies relying on AWS's cloud hosting services such as Netflix, Pinterest, and Instagram. The incident sparked a discussion about the different types of "cloud" services and their sensitivities to failure. Salesforce.com was among the first to promote the benefits of the cloud, offering Software as a Service (SaaS). Heroku, acquired by Salesforce.com, provides Platform as a Service (PaaS) that runs on top of Amazon's AWS service. AWS is Infrastructure as a Service (IaaS), allowing users to rent virtualized hardware resources. Underlying all these cloud services are servers, switches, and routers. When hardware fails in the cloud, different services react differently. Salesforce.com runs their own hardware and software with replication across multiple systems for data and application layers. Replicating data is relatively easy, but keeping it in sync is hard, especially when locations are geographically separated. Data synchronization makes geographic scaling of the Data & Application layer difficult, while the Front End layer can be distributed geographically without special application logic or complex data replication strategies. CloudFlare offers a scalable front end layer that can run in front of any web application to help it better scale and protect against threats and attacks.

HTTPS, the encrypted version of HTTP, is crucial for protecting data sent over the internet but can slow down webpage load times due to SSL's overhead. The main source of SSL performance tax comes from the initial setup of a new connection and the need to negotiate an SSL session between the client and server. CRL and OCSP servers are often not optimized for performance, adding extra time to SSL connections. To address these issues, CloudFlare is working on techniques to reduce the number of connections needed, improving CRL and OCSP response times, and integrating custom-tuned OpenSSL for better encryption performance. Additionally, they have made SSL easy to implement by including it automatically in their paid plans and allowing customers to upload their own certificates.

Mobile web browsing experiences different challenges compared to desktop machines due to varying network conditions on mobile networks. TCP, the protocol used for both types of browsing, performs well on fixed-line Internet but not as effectively on mobile networks. This is because phones are constantly moving and changing their network parameters, which TCP was not designed to handle. Mobile networks also introduce packet reordering issues that can cause additional slowdowns. CloudFlare helps solve these problems by customizing the parameters inside the TCP stacks in its web servers and dynamically determining the best way to behave on a connection based on user's device and network conditions.

Dome9 is a security management service that complements web security provided by CloudFlare. It protects cloud or hosted server firewalls and admin services for all server applications. The service is easy to set up, requiring only a few minutes to create an account, install the agent on the host machine, and establish a security policy. Dome9's Magic IP feature allows traffic only through the CloudFlare network by maintaining special IP address lists automatically. Together, CloudFlare and Dome9 provide enhanced security for cloud or hosted servers with user-friendly next-generation services.

Trumpet is a simple CloudFlare App that allows users to place a short message on top of every page on their website. It was developed by Martin Reistadbakk as a solution for clients who needed to notify users about planned maintenance and downtime. The app is designed to be reliable, small, lightweight, and without any dependencies on JavaScript libraries. Trumpet can show messages across all pages and will not display them again for 24 hours after the user dismisses it. It was originally a Google App Engine project but has since been integrated into CloudFlare's platform, making it faster and more reliable.

Google has proposed a new protocol for web page downloading called SPDY, which aims to speed up browsing without replacing HTTP. The current system uses HTTP over TCP, but SPDY sits between them to enhance the HTTP protocol by altering its interaction with TCP. By allowing multiple requests on a single connection and responses coming back in any order, SPDY minimizes connection set-up cost and slow start while eliminating blocking.

The text discusses the factors that affect the speed of internet connections, particularly between long distances such as San Francisco and London. It highlights that even with a 100 Mbps connection, transferring large amounts of data can take much longer due to latency issues caused by the TCP protocol and the speed of light. The text also explains how CloudFlare helps combat these limitations by having data centers around the world, enabling users to connect to the nearest one for faster browsing speeds.

John Graham-Cumming discusses how storing customer passwords has evolved over time and emphasizes the importance of Kerckhoff's Principle in ensuring security systems are robust. The principle states that a cryptosystem should be secure even if all its details, except for the key, are public knowledge. This is demonstrated through historical examples such as the breaking of the Nazi German Enigma cipher and modern encryption methods like HTTPS, SSL, AES, and RSA. When storing passwords securely, companies rely on the complexity of algorithms like bcrypt, assuming that even in a worst-case scenario where their database is accessed, the security system will still be effective.

The text discusses various methods of storing passwords and their security implications. It begins with plain text storage, which is highly insecure as it allows hackers to easily access all accounts if the database is compromised. Encrypted or hashed passwords offer slightly better security but are still vulnerable to attacks. Hashing using a cryptographic hash function like SHA-1 provides more protection, but can be cracked by using precomputed rainbow tables. Adding salt (a random string of characters) to the hash makes it unique for each user and renders rainbow tables useless. However, as computing power increases, even salted hashes become vulnerable to cracking. To address this issue, slow hash functions like bcrypt have been developed, which can be made slower over time to keep pace with faster computers. The text emphasizes the importance of regular security reviews and choosing long, complex passwords for better protection against hackers.

CloudFlare has started rolling out a beta version of SPDY, a new network protocol designed by Google to speed up web page loading times. The protocol is supported by Chrome and Firefox browsers but not yet by Internet Explorer or Safari. It works by multiplexing multiple objects from a particular site into one request, reducing connection overhead and increasing performance. However, it requires a valid SSL certificate and only works on a per-domain basis. CloudFlare simplifies the setup process for SPDY support and offers additional features like Rocket Loader to enhance its benefits. The beta is being rolled out to select customers with SSL enabled.

The Transmission Control Protocol (TCP) is a fundamental protocol of the internet, responsible for dividing data into packets and sending them across networks. Traditional Congestion Control Algorithms adjust packet size and number in response to network congestion. However, mobile devices have changed how congestion occurs, making traditional algorithms less effective. CloudFlare, with its unique position monitoring global network behavior, has updated its TCP Congestion Control Algorithm to better handle modern wireless connections. This improvement is part of CloudFlare's mission to build a better internet by optimizing underlying protocols.

Over the past 18 months, Cloudflare has been developing a massive network to improve website security and performance across hundreds of thousands of websites. This growth has provided unique insights into these areas that no other company possesses. As a result, Cloudflare is introducing two new tiers of service: Cloudflare Business and Cloudflare Enterprise. These plans offer higher levels of customer support, greater customization, full service level agreements, and advanced technologies such as Cloudflare's Advanced DDoS Protection and Railgun™ Web Optimization. Feedback from businesses and enterprises that have tested these new tiers has been positive, with customers reporting improved performance and security for their websites. Experts in the field also agree that these features are game-changers. Additionally, Cloudflare remains committed to its existing Free and Pro plans, continually investing in improvements and occasionally migrating features from paid plans into the free service.

On June 6th, 2012, Matthew Prince announced the release of Mirage, an automatic image loading management system designed to maximize website performance. Images make up more than 50% of a typical website's data, and tools like CloudFlare's Polish can reduce their size. However, not loading unnecessary images would be even better for performance. Mirage addresses this by only delivering the necessary images based on the visitor's viewport, device type, and network connection. It also optimizes images to the appropriate size and resolution, reducing bandwidth usage and improving mobile performance. While it requires a paid subscription, Mirage can significantly enhance website responsiveness and efficiency when combined with Polish.

On June 5, 2012, Matthew Prince discussed the average web page's increasing data size and the importance of reducing it for faster loading times, especially on mobile devices with limited bandwidth. He introduced a new feature called Polish that optimizes images on websites automatically. The two available modes are Lossless, which removes unnecessary bloat without compromising image quality (21% average file reduction), and Lossy, which applies compression algorithms to suitable images for further optimization (48% average file reduction). While Polish is only available for paid accounts due to added CPU overhead, it significantly improves website performance on wireless devices.

On June 4th, 2012, Matthew Prince reported a hacking incident that targeted his personal Gmail account and subsequently compromised CloudFlare's email accounts. The hack was made possible by four key security flaws: AT&T redirecting voicemail to a fraudulent box, Google's account recovery process being tricked by the fraudulent voicemail, a flaw in Google's Enterprise Apps account recovery process allowing two-factor authentication bypass, and CloudFlare BCCing transactional emails to some administrative accounts. To mitigate these risks, Prince suggests enabling two-factor authentication using apps like Google Authenticator instead of relying on phone networks, removing phone numbers from Google accounts, and not BCCing password reset messages to administrative accounts. The hack lasted less than 2 hours, with the hackers in Gmail for about an hour and a half, and CloudFlare's email accounts for around 28 minutes.

On June 2nd, 2012, CloudFlare announced that they have implemented measures to protect their users from the BEAST SSL attack. The solution involves supporting TLS 1.1 and 1.2 as well as prioritizing RC4 cypher suites. This update not only enhances security but also improves overall SSL performance by approximately 30%. As a result, sites using CloudFlare's SSL now receive an A score from SSLLabs, surpassing the scores of major websites like Google and Facebook. All paid CloudFlare plans include SSL by default, ensuring secure connections for all site visitors.

On June 1, 2012, a hacker accessed a customer's account on CloudFlare and changed their DNS records. The attack was possible due to a compromise in Google's account security procedures that allowed the hacker to access Matthew Prince's CloudFlare email addresses running on Google Apps. The hacker initially gained access to Prince's personal Gmail account by adding a fraudulent recovery email address and then resetting the password. Once inside, the attacker accessed the Google Apps administrative panel for CloudFlare and targeted a specific customer, changing their DNS settings temporarily. The incident began in mid-May when an account request was sent to Gmail for Prince's personal email address. The hacker somehow convinced Google's account recovery systems to add a fraudulent recovery email address to his personal Gmail account. The password used on the personal Gmail account was 20+ characters long, highly random, and not used by Prince on any other services, making it unlikely that it was dictionary attacked or guessed. Once the recovery email address was added, the hacker could then reinitiate the password recovery process and get reset instructions sent to the fraudulent email address. Those instructions were then used to reset Prince's personal email this morning. The hacker was able to use Google's password recovery and have the password reset sent to Prince's CloudFlare.com address, which also uses two-factor authentication. After accessing Prince's CloudFlare.com email account, the hacker accessed the Google Apps administrative panel for CloudFlare and targeted a specific customer, initiating a password reset request for their CloudFlare.com account. The hacker was able to access this account in Google Apps and verify the password reset. At that point, the attacker was able to log into the customer's CloudFlare account and change DNS settings to temporarily redirect the site. Working with Google, Prince regained control of the Google Apps accounts (both his personal Gmail account and his CloudFlare.com account). They reverted the changes made to the customer's account and manually reviewed all other password reset requests and DNS changes. No other CloudFlare.com accounts were accessed or altered. To prevent further attacks, Prince removed his personal email address from any association with CloudFlare and added two-factor authentication to his personal Gmail account. He also recommends that users of Gmail or Google Apps take the following steps: add two-factor authentication to their account, ensure their password is strong and not used on other services, and change any password recovery email to an account that cannot be easily guessed by a determined hacker. Google has since fixed the vulnerability that allowed the hacker to access Prince's personal Gmail account, which involved a breach of AT&T's systems that compromised the out-of-band verification. The FBI arrested most, if not all, individuals involved with the attack on June 26, 2012.