April 2014 Summaries

In April 2014, John Graham-Cumming revealed that Heartbleed could leak private SSL keys through its messages. The Heartbleed Challenge demonstrated this vulnerability within hours of launching. Most people who obtained the challenge server's private SSL key did so by searching for prime numbers in Heartbleed message results. OpenSSL was initially believed to cleanse memory of primes, but further investigation showed that it left copies of these numbers throughout its memory space. This made them vulnerable to Heartbleed attacks. To address this issue, patches were developed and submitted to the OpenSSL team, including one that cleanses memory before freeing it and another that prevents caching of Montgomery parameters. A more radical solution is not storing private keys within OpenSSL at all, which CloudFlare has been testing.

On May 8th and May 21st, 2014, CloudFlare is hosting two meetups at their headquarters in San Francisco. The first one on May 8th will be focused on cryptography with speakers discussing topics such as next-generation protocols for end-to-end security, Firefox Sync, and new developments in Elliptic Curve Cryptography. The second event on May 21st is a Go programming meetup for experienced developers. Both events will provide opportunities for networking and learning, with pizza and beverages provided by CloudFlare.

CloudFlare recently reissued and revoked all of its customers' SSL certificates due to the Heartbleed vulnerability. The decision was made after considering the costs associated with certificate revocation, which can impose significant bandwidth burdens on both website visitors and Certificate Authorities (CAs). When most browsers visit HTTPS websites, they use either Online Certificate Status Protocol (OCSP) or Certificate Revocation List (CRL) to check if a site's certificate has been revoked. OCSP imposes more backend lookups but less bandwidth cost, while CRL generates fewer requests but can become large and bandwidth-intensive. After reissuing all SSL certificates, CloudFlare saw a spike in global CRL activity, with Globalsign's CRL growing from 22KB to approximately 4.7MB. This generated around 40Gbps of net new traffic across the Internet and would have added $400,000 USD to Globalsign's monthly bandwidth bill if they were using AWS’s infrastructure, totaling at least $952,992.40/month. The costs associated with certificate revocation can be significant, and CAs may not always have the resources in place to handle increased load.

On April 17, 2014, Nick Sullivan discussed the Heartbleed vulnerability and its impact on CloudFlare. The company issued a challenge to find out if their private keys were at risk due to this bug. After revoking and reissuing all SSL certificates they manage, they concluded that within two hours a dedicated attacker could retrieve a private key from a vulnerable server. They recommend administrators running servers using vulnerable versions of OpenSSL to patch the software and reissue and revoke all their private keys.

The CloudFlare Heartbleed Challenge has been solved, and the private key for cloudflarechallenge.com was obtained by several authorized attackers using the Heartbleed exploit. Private key holders can impersonate the site, as demonstrated by Fedor Indutny. Although the certificate has been revoked, the site remains active to test browser behavior when encountering expired certificates. Internet Explorer and Safari give warnings but allow users to proceed; Firefox denies access, while Chrome allows loading without warning due to disabled online verification. Certificate revocation rates have increased since Heartbleed's discovery, with more websites evaluating the risk of stolen private keys.

OpenSSL, a widely-used open source library, revealed a major bug known as "heartbleed" on April 11, 2014. The vulnerability allows an attacker to send a specially crafted packet to a vulnerable server running an unpatched version of OpenSSL and retrieve up to 64kB of the server's working memory. There was initial concern that this could expose private SSL keys, making sites vulnerable to impersonation. However, after extensive testing by CloudFlare engineers, it appears that while Heartbleed can reveal sensitive data from HTTP and TLS requests, extracting private SSL keys is extremely hard or possibly impossible on most NGINX servers. Despite this, as a precautionary measure, CloudFlare has begun the process of reissuing and revoking potentially affected certificates.

On April 11, 2014, a Heartbleed Challenge was announced where participants were tasked with stealing a private key from an nginx server using a vulnerable version of OpenSSL. Two individuals independently succeeded in retrieving the private keys using the Heartbleed exploit. The first valid submission came at 16:22:01PST by Software Engineer Fedor Indutny, who sent around 2.5 million requests. The second was submitted at 17:12:19PST by Ilkka Mattila from NCSC-FI, who sent approximately a hundred thousand requests. Two more winners were later confirmed: Rubin Xu and Ben Murphy. All participants used only the Heartbleed exploit to obtain the private key. The challenge highlights the power of collaboration and emphasizes the potential danger posed by this vulnerability.

On April 10, 2014, a security flaw in the popular WordPress plugin Jetpack was discovered. The vulnerability allowed attackers to perform actions on blogs without logging in, such as posting content. This issue has been assigned CVE number CVE-2014-0173 and is fixed in Jetpack 2.9.3. All users of the plugin are advised to update immediately. CloudFlare customers using WordPress are automatically protected against this bug through a Web Application Firewall (WAF) rule, but upgrading Jetpack is still recommended for optimal security.

On April 7, 2014, a vulnerability in OpenSSL 1.0.1 was announced, allowing attackers to reveal up to 64kB of memory to connected clients or servers (CVE-2014-0160). CloudFlare fixed this issue before it went public and all sites using their SSL service were automatically protected. OpenSSL is the core cryptographic library used by CloudFlare for SSL/TLS connections, with a large deployment on the internet. They encourage others running servers that use OpenSSL to upgrade to version 1.0.1g or recompile with the OPENSSL_NO_HEARTBEATS flag enabled for protection against this vulnerability. This bug fix exemplifies responsible disclosure, where stakeholders are given a chance to fix issues before public disclosure, helping keep the internet safe.

Cloudflare has introduced a new feature called CNAME Flattening that allows users to safely use a CNAME record as their root record in the company's DNS without triggering any error conditions. This is particularly useful for those using cloud-based services and prefer not to have a "www" subdomain. The feature enables greater flexibility by allowing users to point their domain to an IP address, even if it changes due to load balancing or other factors. CNAME Flattening has been tested extensively over the last few months and is now available for all Cloudflare DNS users.

CloudFlare, a company known for its fast and secure DNS services, has developed an authoritative DNS server called RRDNS in Go. This server enables them to handle DNS attacks effectively and consistently rank as one of the fastest DNS providers on the web. One less-utilized feature of DNS is the LOC record, which allows users to specify a physical location. Although only 743 LOC records exist in CloudFlare's entire database, they still support this feature. The textual LOC format and the binary, on-the-wire format are described in RFC 1876. To fix an issue with serving LOC records, a parser for the LOC text record type was written and rolled out.