How we used iptables to replicate UDP traffic when upgrading our Graylog cluster

Zapier undertook the challenge of upgrading its Graylog cluster without downtime by creating a parallel cluster to accumulate historical data over a week before decommissioning the old one. To achieve this, they needed to index every log message, sent as UDP datagrams in the GELF format, in both the old and new clusters. To avoid the overhead and complexity of having the logging client send messages to both clusters or using an unreliable UDP proxy, Zapier opted to use iptables on the old Graylog cluster. This approach involved using the TEE target of the mangle table to clone incoming UDP packets and forward them to the new cluster's hosts based on a statistical distribution. The method required enabling IP forwarding and adjusting settings for AWS, but ultimately succeeded without impacting performance or increasing load on the nodes, offering a seamless transition to the updated infrastructure.