Company
Date Published
Author
Rafer HazenRafer Hazen
Word count
1995
Language
English
Hacker News points
15

Summary

We built a scalable telemetry pipeline using Apache Kafka and a sharded PlanetScale database to collect, store, and serve telemetry data for our Insights query performance tool. To power Insights search, we use a hybrid solution that combines Prometheus for database-level aggregations and MySQL for storing query-pattern level statistics and individual slow query events. The Insights pipeline begins in VTGate, which sends aggregate summaries and slow query events to Kafka every 15 seconds. We sharded the Insights database cluster based on customer database ID to ensure scalability and built a custom binary format for reading and writing sketches in MySQL, allowing us to calculate percentiles without pulling underlying data into our application. We chose MySQL as a time series database due to its high cardinality of primary dimension, infrequent changes, ability to filter the dataset, natural shard key, and operational overhead being manageable with sharded Vitess and Kafka.