June 2021 Summaries

Model drift refers to changes in the distribution of a model's input or output data over time, which can lead to decreased performance. It is crucial for ML practitioners to monitor and troubleshoot drift to ensure their models stay relevant, especially in businesses where data is constantly evolving. Drift can be categorized into feature drift (change in input distribution) and concept drift (change in output or actuals). To measure drift, one can compare the distributions of inputs, outputs, and actuals between training and production using various distribution distance measures such as Population Stability Index (PSI), Kullback-Leibler divergence (KL divergence), and Wasserstein's Distance. It is essential to relate these metrics to important business KPIs and set up thresholding alerts on the drift in distributions. When a model drifts, retraining it may be necessary, but careful consideration of how to sample new data and represent it in the model is required to avoid overfitting or under-representation. Adjusting the tradeoff between these two competing forces can help strike the right balance. If significant changes have occurred in the business, a simple retrain might not suffice, and the entire model may need revision. Troubleshooting drift involves identifying which input features or outcome variables have changed, understanding how their distributions have shifted, and potentially adjusting the model structure or feature engineering to adapt to new dynamics. Regularly reviewing model performance and maintaining open communication with end-users can help proactively address drift issues and improve models over time.

Arize AI has grown and welcomed new team members, including Andy Lu as Senior Designer, Harrison Chu as Senior Software Engineer, Krystal Kirkland as Product Marketing Manager, David Monical as Application Engineer, Eric Senzig as Enterprise Sales Lead, and Francisco Castillo as Data Scientist. These individuals bring diverse backgrounds and experiences to the company, with a focus on building a brighter future in the AI/ML space. Arize's strong stance on diversity, fairness, ethics, and inclusion has fueled Andy Lu's passion for the role, while Harrison Chu aims to solve operational challenges faced by ML-based teams. Krystal Kirkland is excited about contributing to a team that prioritizes technology and society, while David Monical was drawn to the company's potential to benefit from its MLOps capabilities. Eric Senzig appreciates Arize's unique mix of people, technology, and market timing, and Francisco Castillo joins with the goal of holding AI accountable through transparent and ethical model observation.

The growing importance of responsible AI was discussed by Arize AI CPO Aparna Dhinakaran at Re-Work, highlighting challenges such as lack of access to protected attributes, no easy way to check for model bias, tradeoff between fairness and business impact, and responsibility diffused across individuals, teams, and organizations. The presentation emphasized how human bias can be introduced into data through proxy information and sample size data, leading to biased models that disproportionately affect certain groups. To optimize model fairness, Aparna recommends increasing organizational investment, defining an ethical framework, and establishing tools for visibility, such as ML observability, which enables the identification of problems before deployment and allows for troubleshooting and fixing issues.

UbiOps and Arize are partnering to accelerate model building and deployment by providing a comprehensive observability platform for machine learning engineers, enabling teams to develop models, deploy them to production, and gain full visibility and control of their performance. This integration allows data scientists and ML engineers to work together seamlessly, validating model quality and performance prior to deploying to production, accelerating model deployment without high ops overhead, and automatically diagnosing issues that emerge in production. The platform provides features such as performance heatmaps, enabling teams to find opportunities for improvement and retraining, ultimately delivering better model performance and faster time-to-value.