March 2023 Summaries

Webhooks are becoming an essential requirement for SaaS consumers when making purchasing decisions, offering numerous benefits such as workflow automation and no-code integrations, increased usage, improved customer experiences, reduced churn, and a competitive edge. By integrating webhooks into their product-led growth strategy, businesses can drive growth, improve user engagement, and differentiate themselves from competitors.

We suffered a partial outage on Saturday the 11th of March due to OOM errors in our containers, which resulted in most API calls returning 5xx errors in the US region. The issue was not caused by code changes or unusual traffic, but rather an unexpected memory usage spike that led to repeated container killings. We suspect an edge case with a library used for serialization may be responsible, and have since reproduced the issue locally and mitigated it using a different allocator. The incident highlights the importance of monitoring and understanding AWS metrics, as well as the need for additional observability tools. Our team is taking steps to prevent similar outages in the future, including increasing memory capacity and using more fragmentation-resistant allocators.

Receiving webhooks is crucial for services to notify each other of events. Webhook endpoints can be vulnerable to security threats, such as attackers impersonating services by sending fake webhooks. To mitigate this, most webhook implementations sign their messages, allowing receivers to verify them using digital signatures. Svix offers an open-source library to simplify the verification process. This tutorial demonstrates how to set up a secure and scalable webhook endpoint with Netlify Functions that verifies Svix webhook signatures automatically, eliminating common failure modes and making it easy for developers to focus on their business.

SSRF attacks occur when an attacker coaxes a server into returning sensitive information residing on the server itself or in its internal network. To protect against SSRF attacks, Svix recommends a multi-tiered approach to security, including restricting applications' access to only needed internal resources and using subnet filtering, which allows users to whitelist specific IP addresses or subnets to prevent requests to unauthorized resources. The open-source Svix platform provides protection out of the box, but users can also configure this feature to customize their settings, such as disabling SSRF protection entirely if necessary.