Company
Date Published
Author
Andrew Cholakian • Jordan Sissel
Word count
1112
Language
-
Hacker News points
None

Summary

The blog post discusses the evolution of Logstash output workers from version 2.1 through the upcoming 2.4 and 5.0 series, highlighting changes in performance and concurrency handling. Initially, Logstash had slow and safe defaults, which shifted to faster but less stable settings in version 2.2, causing some confusion and issues due to the introduction of the NG Pipeline. This version integrated Filter Workers and Output Workers into a single Pipeline Worker system, which altered the threading model and required users to adjust output worker settings for optimal performance. The post further explains the challenges faced when attempting to autoscale output workers, as many plugins were not thread-safe, leading to the decision to revert to a single output worker in subsequent versions. Looking ahead, Logstash aims to eliminate output workers entirely, adopting a 'shared' output model where concurrency is managed internally by each plugin, starting with the Elasticsearch output, to ensure both performance and correctness. Users are encouraged to consult the new Performance Tuning Guide and participate in community discussions for further improvements.