400 Errors When Migrating from NGINX to Envoy

After transitioning from NGINX to Envoy, a significant increase in HTTP 400 errors was detected, primarily affecting POST endpoints, due to requests missing payloads. The Core Infrastructure Team investigated by capturing network traffic and discovered that the client was closing connections prematurely, preventing the payload from being sent, which resulted in 400 errors. Unlike NGINX, which buffers requests before sending them to the backend, Envoy streams requests, leading to the issue. The solution involved activating Envoy's buffer filter option, allowing it to wait for the entire request before forwarding it to the backend, effectively resolving the errors. This investigation not only addressed the immediate problem but also highlighted a potential issue with clients sending incomplete requests, prompting further investigation to ensure reliable service. The process underscored the team's commitment to reliability and the value of deep technical investigations.