August 2022 Summaries

At Tecton, a platform engineering team is crucial to ensure feature correctness and accuracy in their customers' data stack. The team uses canary testing as a key part of their test-first approach to development, which involves slowly rolling out changes to a small subset of users before making them available to all users. This approach helps prevent unstable or buggy releases from entering production. Tecton's canary pipeline provides signals on whether the latest changes impact customers' materialized feature data, allowing the team to measure and ensure accuracy. The pipeline is designed to be cost-effective, reliable, and easy to distinguish between legitimate differences in feature values and false positives. It also handles streaming jobs, which are unpredictable due to sporadic data arrival and lagging partitions, by using a "best effort" approach that gives a good approximation and works well across different streaming sources and platforms. The canary process has helped detect day-to-day regressions, roll out large changes with high possibility for breakage, and prevent rolling regressions into production, ultimately providing a better customer experience.

Building real-time data pipelines for machine learning is challenging due to the need for fast access to feature data, maintaining standing infrastructure, and handling fresh features from multiple sources. The process typically starts with batch feature engineering using tools like data warehouses, data modeling tools, and schedulers, but online inference adds complexity by requiring precomputed features stored in a fast database like Redis. Fresh features multiply the amount of infrastructure needed to manage, and training/serving skew can occur when features are computed in two distinct places. Feature platforms like Tecton provide tools to centrally build and manage diverse data pipelines for machine learning models, helping teams avoid these challenges and simplify the development of real-time data pipelines.

This post demonstrates how to build a real-time machine learning system using Tecton and Databricks, simplifying the challenges of building operational machine learning systems that require real-time data. By leveraging Tecton's feature platform and Databricks' MLflow integration, teams can create a model serving endpoint in 15 minutes or less, including real-time data processing and online inference. The system is designed to handle high latency constraints and ensure feature freshness, making it suitable for applications that require fast and accurate predictions. By using Tecton and Databricks, developers can build real-time ML systems without months of manual engineering work.