Why am I seeing bulk rejections in my Elasticsearch cluster?

In the blog post by Christian Dahlqvist, the issue of bulk rejections in Elasticsearch clusters is addressed, focusing on the causes and potential solutions. Bulk rejections occur when the indexing workload surpasses the cluster's capacity, leading to parts of the bulk requests being rejected to protect the cluster from overload. The post explains the process of handling bulk requests and the role of the coordinating and data nodes. It highlights that increasing the bulk queue size is not a viable solution as it may result in cluster instability due to increased memory usage. Testing with different cluster configurations shows that adding more data nodes reduces rejection rates by distributing the load more evenly. The post advises against merely enlarging the bulk queue size and instead suggests understanding the source of rejections, ensuring even load distribution, and potentially scaling the cluster. It also emphasizes the importance of handling rejected documents properly by inspecting responses and retrying as necessary, as done by tools like Logstash and Beats.