August 2022 Summaries

Sequin initially used fixed pricing for subscriptions in its Stripe configuration, which worked well until it became evident that static prices didn't adequately reflect the value delivered or support costs for each customer. To address this, Sequin implemented metered billing, where customer bills are calculated based on product usage, specifically using monthly active rows (MAR) as a metric. However, managing metered billing posed challenges, as it required frequent reporting of customer usage to Stripe, which complicated the process due to the need for multiple API requests. To streamline this, Sequin developed a sync with Stripe that replicates Stripe API data, including Customers, Subscriptions, Subscription Items, and Prices, into a Postgres database. This synchronization allows for a simplified billing process where a single database query replaces multiple API calls, effectively reducing HTTP requests and improving efficiency. The system now automatically posts usage reports using a cron-like job, making it robust and predictable while simplifying the application’s metered billing management.

In a deep dive into optimizing resource allocation using Elixir's Observer, Sequin's team explored the challenges of dealing with machine overload despite moderate CPU utilization, highlighting how Observer helped diagnose the issue. The team used Observer to examine the scheduler's run queue and process attributes to identify bottlenecks, such as high scheduler utilization and large message queues, which indicated an overloaded machine. They learned that Erlang processes are lightweight and managed by the BEAM, which distributes CPU core time among processes efficiently, but issues can arise when processes have to wait for CPU time. To mitigate these issues, they optimized process memory usage and rearchitected processes handling excessive messages, improving system performance. Additionally, they ran experiments with the Flow library to parallelize CPU-intensive tasks, discovering the importance of cautious resource allocation to prevent starving other processes in the system. Overall, the insights gained from Observer allowed the team to better manage their system's runtime properties and prepare for future challenges.