January 2022 Summaries

Stefan Kalb, CEO of Shelf Engine, discusses his company's mission to eliminate food waste and revolutionize the grocery business through AI-driven technology. Founded in 2016, Shelf Engine has diverted over 4.5 million pounds of food waste from landfills and helped clients achieve an average gross margin dollar expansion of more than 15%. The company's unique Results-as-a-Service (RaaS) model directly takes on inventory risk and guarantees outcomes for its customers, including major retailers like Kroger, Target, and Whole Foods. Kalb highlights the persistent problem of product availability, quality, and consistency in grocers due to supply chain disruptions and emphasizes Shelf Engine's ability to address these issues through AI-driven order automation. The company also tackles shrink discrepancies, which occur more than 75% of the time for large and small grocers, by providing accurate waste data that helps retailers understand their true shrink problems. In response to COVID-19's impact on businesses, Shelf Engine assists customers in adapting by relieving pressure on labor through AI-driven ordering systems. The company's success is attributed to its highly capable AI team and the unique advantage of capturing data from stores and vendors that helps inform accurate predictions for each store's performance. Lastly, Shelf Engine uses Arize AI for model monitoring and ML observability, which provides valuable insights into error metrics, lineage, versioning, and proactive detection of model drift. The company plans to scale its technology further in the coming year to work with more retailers and help them capture growth opportunities.

Aman Khan, Arize's newest product manager, brings experience in pioneering ML infrastructure and tooling from his roles at Cruise and Spotify. He will help drive product development of Arize's rapidly-growing ML observability platform, partnering closely with the marketing team to ensure clear customer communication and streamlining sales and customer success processes. With a background in mechanical engineering and a passion for building and coding, Aman has transitioned into an ML/PM role, advising companies on the build-versus-buy calculus for model monitoring and observability, highlighting the importance of load testing and scalability in real-time applications. He offers advice to those starting their careers, emphasizing the need to play to one's strengths and finding a supportive organization with leaders who enable growth.

AUC (Area Under the Receiver Operating Characteristic Curve) is a widely used metric in machine learning that measures the degree of separation between positive and negative classes in a dataset, calculated as the area under a staircase-like curve generated by varying threshold values for prediction scores. It's useful across various use-cases, particularly when models output scores, providing a single-number heuristic of how well a model can differentiate data points with true positive labels from those with true negative labels. AUC ranges from 0 to 1, where 1 indicates perfect separation and 0.5 suggests no separation, and it's often used in data science competitions and when accuracy is insufficient. However, AUC may not be the best metric for all problems, especially those involving probabilities or business outcomes, as it doesn't account for calibrated predicted probabilities or false positive rates. Ultimately, understanding the tradeoffs of using AUC and other model metrics is crucial for selecting the right metric to evaluate a model's performance in a specific context.

To ensure the long-term success and sustainability of AI initiatives, companies should focus on five key areas: diverse teams and representative datasets; ethical and risk governance frameworks for AI; modernized data policies granting access to protected data where needed; monitoring and troubleshooting ML model performance in real-world scenarios; and growing internal visibility, opening the black box, and quantifying AI ROI. By addressing these areas, companies can balance the power and potential peril of AI, maximizing positive outcomes for customers and society at large.

Remi Cattiau is the Chief Information Security Officer (CISO) at Arize AI, a company currently hiring over ten positions. With nearly two decades of experience in cloud security for large enterprises, Remi is responsible for ensuring high standards for Arize's security posture and safeguarding customer data. His career journey includes working with open source projects, leadership roles at startups, and consulting for companies on building out their cloud security. Remi believes that machine learning is trending within the security industry and highlights Arize's ability to help teams visualize and address problems in model monitoring and observability.

America First Credit Union is leveraging machine learning (ML) observability to stay ahead in a competitive market by prioritizing speed and model monitoring. The credit union's Data Science Manager, Richard Woolston, emphasizes the importance of identifying proxy metrics such as drift, delinquency, and fair lending regulations to ensure portfolio health and mitigate bias. Arize AI's platform helps America First Credit Union troubleshoot issues and make data-driven decisions by providing automated alerts, feature slicing, and collaboration with product teams. As ML models become increasingly prevalent in lending, Woolston envisions a future where access to credit becomes easier for traditionally excluded groups and regulations become more streamlined through self-documenting systems.

Customer lifetime value (LTV) is a crucial metric to evaluate a company's overall sales motion, especially in non-contractual sectors like consumer packaged goods or retail. LTV models predict future purchasing behavior and help increase profitability by identifying valuable customers. These models use machine learning algorithms to analyze patterns of engagement based on predictions. Monitoring and observability are essential for LTV models as they must iterate and quickly estimate long-term value with delayed or no ground truth data. ML observability platforms should set up baseline monitors, evaluate feature, model, and actual/ground truth drift, and measure model performance to improve overall business outcomes.