July 2014 Summaries

Redis is a popular modern database option that has aided in creating over 50,000 databases for its 2,500 paying customers, with more than 100 new databases created daily. It is widely used in various industries such as social applications, online advertising companies, and gaming companies. However, running Redis in the cloud can cause performance degradation due to its fast response times, which require multiple instances to be incorporated into a cluster to enable pure multi-tenancy architecture without degraded performance. To address this challenge, users need to consider data center selection carefully to avoid network latency, and instance selection should not be limited by predefined sizes, allowing flexibility in terms of size and type. Additionally, data persistence is crucial for high-performance applications like those running 1 million operations per second, which can be achieved through hybridizing ephemeral storage with EBS infrastructure. Monitoring is also essential, where tools like Zabbix and Limbic are used to monitor nodes and database metrics respectively. A strong DevOps team plays a crucial role in handling complex infrastructures, using a 'baby steps' approach to move from manual configuration to automation.

Redis, an in-memory database, uses client buffers to serve client requests, which can be a source of headaches if not managed properly. Client buffers are allocated memory space for each connection and store response data before it's read by the requesting client over the network. If these buffers become too large, they can cause Redis to run out of memory, leading to eviction policies being triggered. This can happen even with top-performing clients and network links, and certain commands like KEYS, SMEMBERS, HGETALL, LRANGE, and ZRANGE can be particularly problematic if used irresponsibly. By setting client buffer thresholds accordingly and using the SCAN family of commands instead, developers can prevent memory-related issues and maintain a smooth Redis experience.

The Memcached Cloud Plugin for WordPress has been released, implementing the WordPress Object Cache based on the PECL Memcached extension and libmemcached library. The plugin offers advanced features such as multi set and get methods, 'by key' functions, and SASL authentication mechanism for secured connections to Memcached servers. To simplify the installation process for users deployed in untrusted networks or requiring more control over clients connecting, the plugin allows configuration of buckets with SASL effortlessly. The plugin is designed to work seamlessly with Heroku's PHP environment, supporting third-party extensions like Memcached through composer.json file. By following a step-by-step guide, WordPress users can deploy their application to Heroku and utilize Memcached Cloud for enhanced performance.

Redis provides a wide variety of tools to improve efficient in-memory database usage, but misconfiguration can lead to operational challenges and performance issues. The replication process is particularly prone to issues, with the initial synchronization stage being trickier to complete than the ongoing stage. This can cause the replication timeout to be reached, leading to repeated attempts at synchronization and log messages indicating partial or full resynchronization not possible. To address this, it's essential to estimate the time needed for replication based on factors such as storage speed, dataset size, and network performance, setting a suitable replication timeout value that takes these factors into account. Regular review of the timeout setting is also crucial as databases grow. By understanding and configuring Redis' replication settings effectively, developers can ensure their database remains available and scalable.

We use Redis as our caching layer due to its set operations being useful for the logic in our API service. However, we've encountered scaling issues with Redis, primarily due to its single-threaded nature and limited horizontal scaling capabilities. We initially struggled with memory usage, but disabled saves to utilize all available memory on the box. Our current bottleneck is approaching the upper bound of available CPU. To overcome this, we attempted to use Redis clustering, but were limited by the open-source version's status as a work-in-progress. Instead, we employed blue-green traffic testing using em-proxy, which allowed us to test our new cluster with live production traffic. The results showed significant performance improvements, with response times averaging 55ms compared to intermittent spikes of up to 700ms with the standard open-source Redis server. We discovered that disabling certain commands and adjusting key naming conventions were necessary to resolve issues related to sharding and object management. Our testing demonstrated the powerful nature of this method and highlighted the potential for significant performance gains by utilizing Redis, particularly when compared to our prior environment.

The author created a web toy using Redis' commands and D3.js, inspired by Mike Bostock's example. The tool groups Redis' 160+ commands by topic and allows users to zoom in/out using the mouse. Each command is represented as a circle with size corresponding to its time complexity, and clicking on it opens its documentation page from redis.io.

The Redis team has open-sourced a high-throughput benchmarking tool for Redis and Memcached resources called memtier_benchmark. The latest version of the tool includes several improvements, including the generation of randomized data according to known size range using the --data-size-pattern switch, the ability to use Gaussian distribution to access the data with options such as key-stddev and key-median, and the support for using SETRANGE and GETRANGE Redis commands. These additions aim to better mimic real-life use cases and provide more flexibility in benchmarking.