Company
Date Published
Author
Marco Salazar
Word count
1592
Language
English
Hacker News points
51

Summary

Marco Salazar, the author of this text, is a developer working on Dagster, an open-source tool for building and running data pipelines. Recently, their cloud product was launched, leading to increased usage by new and existing customers, which resulted in some new and exciting problems. The log viewer, a critical component of the application, wasn't loading for one customer due to a massive influx of log lines per second. This issue led Marco and his team to investigate how the log viewer works and triage the problem. They discovered that the client CPU was pegged at 100% due to parsing and rendering competing for the main thread. To solve this, they implemented web workers, which significantly reduced CPU utilization. However, upon further investigation, they found that using a `cursor` variable in the subscription hook caused issues with subscription tearing down and re-creation. Solving these problems involved restructuring queries to not include the `cursor` variable and moving their use of `useSubscription()` into a component that returned null. The root cause fixes were deployed without introducing web workers, resulting in improved performance and user satisfaction. The team learned valuable lessons about profiling, intuition, solving for one problem causing another, and using external resources like Google and Stack Overflow.