Solr: Improving performance for Batch Indexing

The narrative discusses improvements made to Box's 'full' indexer, a system designed to create a search index by reading documents from an HBase table and inserting them into a Solr index. Initially, all mapreduce jobs communicated with all shards, which sometimes led to bottlenecks when Solr's in-memory structures flushed to file, causing temporary slowdowns. To address this, the author first attempted to increase the worker pool to prevent workers from being stuck on slow shards, but this was insufficient. A subsequent strategy involved creating separate queues and workers for each shard within each mapper, which significantly improved system flow by allowing other shards to continue processing even if one was slow. While this new model enhanced performance—doubling the number of reads per second and enabling the indexing of nearly 50 billion documents in less than two days—it also introduced drawbacks, such as the lack of communication between workers targeting the same shard and increased memory demands that could limit scalability. A more scalable solution would involve a queue between mappers and Solr shards, with shard-specific clients to manage document indexing.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.