Monitoring A/B Experiments In Real Time

We have implemented a real-time experiment metrics pipeline using SingleStore, which provides efficient and scalable data processing capabilities. The dashboard offers pre-computed hourly metrics for the past hour, as well as real-time metrics within the last hour, allowing us to quickly analyze and validate triggering issues, group size changes, and core metric performance. This system has been in production for over six months and has already helped us in numerous situations, providing valuable insights into experiments and enabling us to catch bugs and avoid disastrous changes early. We use Streamliner, an integrated SingleStore and Apache Spark solution, to ingest data from Kafka and persist it into SingleStore, ensuring low latency for near real-time analysis. The system is built using Python libraries provided by SingleStore, allowing us to connect to the database and run scheduled jobs or on-demand queries.