Discovering Anomalies in Real-Time with Apache Flink

Mux, a video streaming platform, developed an anomaly detection system in Summer 2016 using Apache Flink. The system was designed to scale easily without sacrificing performance, support alerting for infrequently-played videos, and have low latency. Apache Flink is a stream processing engine that can handle unbounded event streams with guaranteed exactly-once event-time processing. It allows real-time Map-Reduce operations on streaming events and supports both processing-time and event-time modes for time-series data. Mux uses Flink to calculate playback error rates across video titles and customer properties, enabling real-time alerts and anomaly detection.