June 2023 Summaries

The key to unlocking the true potential of webhooks lies in providing comprehensive documentation that covers all aspects of event triggers, schema, testing, security measures such as signature verification and IP whitelisting, error handling, retry schedules, and more. Even with a secure and reliable webhook service, poor documentation can lead to a seamless user experience, whereas well-documented webhooks enable developers to navigate through complex systems effectively. By including real-world examples, code samples, and clear instructions, webhook providers can transform their services into intuitive tools that make using the product a breeze.

The key to providing an easy webhook integration experience for your users lies in accurately defining their event types, which should be considered as part of the API and documented using an OpenAPI specification. This allows users to quickly import and update event types with minimal effort, while also ensuring accuracy and consistency.

The event catalog of Svix, a service that provides webhook functionality, has been made publicly accessible through custom domain hosting. Previously only accessible from the application portal, customers can now host their event catalogs on a custom domain for easier integration with external documentation systems. To make this feature available, users must enable the "Make Public" setting and update their DNS settings to point to `event-types.svix.com`. Once activated, customers will be able to serve their event catalogs from their custom domain across all environments.

** The Omniqueue crate is a high-level queuing system that allows users to focus on sending and receiving types, without worrying about the backend used. It provides a simple and flexible way to build queue producers and consumers, with support for multiple backends such as RabbitMQ, Redis streams, SQS, and an in-memory queue. Users can customize the behavior of the queue by using feature guards, constructing the queue with specific configurations, and defining custom encoders and decoders. The crate is designed to be adaptable to different use cases, including those where the backend may not be known until runtime. With its release, Omniqueue becomes a standalone project that other developers can use, and it will continue to evolve with new features and improvements.

The outage occurred on June 6th, 2023, due to a change made to the database authentication token fetching code, which caused a full table scan instead of using indices in the US region, leading to elevated errors and CPU usage. The issue was only present in the US region, not affecting other regions. An investigation revealed that the problem required a large number of parallel API calls with different authentication tokens to trigger, while continuous canary tests only emulated a small number of customers. The team is still investigating why the indices weren't used and has implemented measures to prevent similar issues, including a company-wide feature freeze, improved deployment speed, and automatic multi-region fallbacks. The incident highlights the importance of thorough testing and monitoring in ensuring the reliability and performance of API services.