Company
Date Published
Author
-
Word count
1740
Language
-
Hacker News points
None

Summary

In environments where fully separating production and pre-production systems is impractical due to resource constraints, using Logstash with UDP provides a viable solution for safely routing a subset of production data to pre-production clusters without affecting the production data flow. This approach leverages UDP’s fire-and-forget nature, which allows data to be sent without needing acknowledgment from the receiver, thus avoiding disruptions even if the pre-production cluster faces issues. Unlike the more complex output isolator pattern that requires a persistent queue, this method uses a Ruby filter for efficient sampling of production events, ensuring only a random subset is forwarded, thereby reducing operational risks and performance overhead. This lightweight solution facilitates testing in pre-production with real production data while maintaining an uninterrupted production pipeline, though it is important to note the inherent unreliability of UDP in guaranteeing delivery.