November 2024 Summaries

In the realm of SaaS products, implementing an events table becomes essential for tasks such as providing status updates, audit logs, and aggregate metrics, often leveraging time-series data segmented by resource IDs. While startups are increasingly adopting specialized OLAP stacks like Kafka and Clickhouse for managing large volumes of events, the text argues that using Postgres may suffice for longer than anticipated. The text delves into three common use-cases for event storage seen in the development of Hatchet: state transitions, aggregate data, and logs, each with its own challenges and solutions for scaling under higher loads. For state transitions, a basic events table with minimal indexing is advised to reduce write overhead, while bulk inserts via buffers are recommended for high-volume writes. Aggregate data, which is sensitive to use-case variations, is effectively managed with TimescaleDB, a Postgres extension for time-series data, allowing for efficient continuous aggregates and partitioning. Logs, akin to state transition events but with a focus on complex searches, can also benefit from Timescale's capabilities, offering performant full-text searches when queries are time-bound. The discussion concludes by advocating for sticking with a primary database like Postgres for simplicity and maintainability, highlighting its advantages in terms of consistent tooling, ease of upskilling, and straightforward data extraction, all exemplified by Hatchet's use of Postgres for task queue visibility.