June 2021 Summaries

Feature logging is a technique used to generate historically accurate training data for real-time machine learning systems by fetching new feature values online at inference time and logging them offline for later use. This approach eliminates the need for complicated time-travel queries and joins logic, making it easier to synthesize accurate training sets. Feature logging also provides additional benefits such as auditing and explainability of models, custom analysis, and data quality monitoring. Tecton, a platform that enables real-time machine learning, has built-in feature logging capabilities that can be enabled with a single parameter, allowing users to log requests and access the logged dataset for model training or analysis.

Real-Time Aggregation Features for Machine Learning (Part 1)` discusses the technical challenges of serving rolling time window aggregations in real-time ML applications at high scale with low latency and high feature accuracy. The main challenges include memory constraints, backfilling historical data, maintaining high feature freshness, and generating training datasets. A naive implementation using a transactional database is not sufficient for high-scale applications, and precomputing aggregations in real-time as new raw data becomes available may also pose technical challenges. To address these issues, companies often employ stream processors like Apache Spark or Flink to run streaming time window aggregations, but this can be limited by memory constraints and the need for backfilling historical data. Maintaining high feature freshness is crucial, but this can be challenging due to the limitations of sliding time windows and the need for a separate compute path to train models using offline feature values.

Tecton's solution for real-time aggregation features for machine learning uses a tiled time window approach, where the aggregation is broken down into compacted tiles of smaller time windows that store aggregations over the tile interval and a set of projected raw events at the head and tail of the aggregation time window. The configuration requires selecting and projecting raw events using SQL, defining the aggregation information using a simple DSL, and streaming ingestion to the online store. Batch ingestion is used for backfilling and forward-filling the offline store. Compacted data is produced by periodic batch Spark jobs that read from the streaming source's offline mirror, reducing the worst-case number of rows that have to be fetched from the store. The solution provides benefits such as ultra-fresh features, compute and memory-cost efficient processing, fast feature retrieval, and scalable storage requirements. Airbnb and Uber use Tecton's implementation for several years. The approach can be generalized beyond just time-window aggregations and extend to completely user-defined transformation steps.