March 2023 Summaries

Sequin offers an efficient solution for syncing data from HubSpot's API to local databases and systems like Kafka, overcoming challenges such as change detection and rate limits. HubSpot's API, a key tool for companies managing marketing and customer service, presents difficulties due to its lack of webhooks for custom objects, incomplete support for associations, and eventual consistency issues in its Search API. Sequin's sync process involves schema introspection, backfilling historical data, and ongoing change detection through strategic API querying, despite HubSpot's limitations in detecting deletes and managing associations. The system employs an in-memory cache to handle inefficiencies and potential data type issues, ensuring that changes are captured accurately even with the API's eventual consistency. By utilizing OAuth apps for dedicated rate limits, Sequin minimizes interference with other integrations, while maintaining real-time data propagation. The sync process abstracts API complexities, reduces latency, and increases data queryability, significantly enhancing operational efficiency for users.

The article explores the BREAK menu feature within the Elixir REPL, IEx, which is part of the Erlang Virtual Machine (BEAM) and allows users to interact with a running Elixir session. It details how pressing Ctrl-C twice in IEx brings up the BREAK menu, providing options such as aborting the session, obtaining crash dumps, continuing the session, and retrieving various system information. These options are also available when running other Erlang applications and shells. The menu's commands, like checking memory usage, listing processes or open ports, and killing processes, are particularly useful for debugging and managing Elixir/Erlang applications, especially when more advanced tools are unavailable. The article offers insight into the implementation and purpose of each command, emphasizing the utility of the BREAK menu for maintaining and troubleshooting applications running on the BEAM.

Sequin, a company that streams data from services like Stripe to databases and messaging systems, encountered challenges with eventual consistency while using Stripe's /events API endpoint for real-time data synchronization. The company initially relied on polling for stronger consistency guarantees over webhooks, but faced difficulties with cursoring through events due to the API's eventual consistency, which can cause missing events when querying rapidly. Stripe's thoughtful change detection features, such as the /events endpoint, allow users to track changes like customer creation or subscription updates. However, the Sequin team discovered that events could be skipped if multiple events were created at the same timestamp, as the API's event IDs are not strictly ordered. To address this, the team implemented a new cursoring strategy that delays updating the cursor if recent events are detected, thus reducing the risk of missing events at the cost of slight latency. This situation highlights the complexities of managing eventual consistency in APIs and the importance of designing predictable and consistent data interfaces for engineering teams.