Company
Date Published
Author
Alex Guziel
Word count
2185
Language
English
Hacker News points
None

Summary

At Tecton, a platform engineering team is crucial to ensure feature correctness and accuracy in their customers' data stack. The team uses canary testing as a key part of their test-first approach to development, which involves slowly rolling out changes to a small subset of users before making them available to all users. This approach helps prevent unstable or buggy releases from entering production. Tecton's canary pipeline provides signals on whether the latest changes impact customers' materialized feature data, allowing the team to measure and ensure accuracy. The pipeline is designed to be cost-effective, reliable, and easy to distinguish between legitimate differences in feature values and false positives. It also handles streaming jobs, which are unpredictable due to sporadic data arrival and lagging partitions, by using a "best effort" approach that gives a good approximation and works well across different streaming sources and platforms. The canary process has helped detect day-to-day regressions, roll out large changes with high possibility for breakage, and prevent rolling regressions into production, ultimately providing a better customer experience.