Using Sentry Logs to Debug a Dynamic Sampling Issue

Sentry Logs played a crucial role in helping the Sentry team debug a complex issue causing transaction spikes in their internal project. The problem arose from intermittent spikes and a 100% sample rate, which led to the project abuse layer discarding large amounts of data. Initially, it was believed that the dynamic sampling configuration was hitting a fallback case, but detailed logging revealed otherwise. By analyzing logs, the team discovered that the issue was not in their code but in the data fed to it, specifically a ClickHouse query with a granularity misalignment that returned zero at the top of the hour. The problem was traced to a race condition caused by the granularity of the query, which was resolved by adjusting it to 60 seconds. This investigation highlighted the effectiveness of Sentry Logs in debugging complex interactions between multiple systems, allowing the team to use their own tools to enhance Sentry's functionality.