Company
Date Published
Author
Guy Boertje • Aaron Mildenstein
Word count
1170
Language
-
Hacker News points
None

Summary

Logstash utilizes message queues like Redis to manage event surges and prevent slowdowns in systems such as Elasticsearch. Redis is favored for its speed and efficiency, having been used since Logstash version 1.0.4. The performance of the Logstash Redis input plugin can be optimized by adjusting the number of threads, which significantly affects throughput. Increasing the batch_count beyond the default of 125 can reduce performance due to changes in the Logstash pipeline architecture. The Redis input plugin now uses batch mode by default, with Lua scripts facilitating batch retrievals, but larger batch sizes can negatively impact performance due to transaction waits. Tuning the Redis input plugin, pipeline batch size, and worker threads is crucial to ensure the input can keep pace with the filter/output stage. Using multiple Logstash instances with a single Redis server can improve throughput, though connection contention should be managed. Future enhancements include integrating a JRuby wrapper for the Jedis library to support Redis Cluster mode.