September 2022 Summaries

AUC (Area Under the Precision-Recall Curve) is a metric used to evaluate the performance of a classifier, particularly useful for imbalanced datasets where the positive class is rare. It measures the degree of separation between positive and negative classes based on their prediction scores. A perfect PR curve would have an AUC of 1, indicating ideal performance. However, PR AUC fails if teams misclassify the positive class or don't consider True Negatives equally with False Positives. In contrast to ROC AUC, which is more suitable for balanced datasets, PR AUC emphasizes precision and recall on the positive class, making it a better choice for tasks like disease diagnosis or fraud detection where identifying minority events is crucial. Understanding the tradeoffs of different metrics is essential when optimizing model performance.

Arize AI and OpenAI are collaborating to help organizations build and deploy unstructured models, such as natural language processing (NLP) models, more efficiently. Unstructured data is a significant challenge for deep learning, requiring human labeling or annotation to group the data and find trends and insights. OpenAI provides AI building blocks like GPT-3, Codex, and DALL-E, which can generate text or code completion, while Arize is an ML observability platform that enables teams to log structured and unstructured data to detect and resolve model performance issues faster. By combining OpenAI's generative models with Arize's logging capabilities, organizations can monitor and troubleshoot their unstructured models in production, reducing costs and maximizing performance. The collaboration allows teams to proactively identify when their data is drifting and troubleshoot using interactive visualizations like UMAP.

A centralized machine learning (ML) team's purpose is to provide a unified and standardized experience for ML application development, freeing data scientists from creating new tools and processes from scratch. However, debates on ideal team structures are heating up, with some arguing that centralized ML teams are falling out of favor due to the emergence of MLOps and decentralized ML approaches. The author, who has built and scaled ML teams, argues that central ML can still be effective if done right, emphasizing the importance of a hybrid organizational structure, preemptive tooling development, and overcoming common challenges such as tooling lock, getting projects on the roadmap, and fence creation. Successful central ML teams share characteristics like having a centralized component and individual engineers on customer teams, building preemptively, and understanding customer pain points. Ultimately, the key to successful central ML lies in organizational culture, mindset, and how you build matters, with a focus on automation, standardization, and collaboration between subject matter experts.

Arize AI co-founder Aparna Dhinakaran and Monte Carlo CTO Lior Gavish discussed the evolving relationship between data and machine learning (ML) infrastructure. They highlighted key differences between data observability and ML observability, emphasizing that both are necessary for modern data practices. Observability goes beyond monitoring by enabling teams to understand why problems occur and how to resolve them. Building trust in data and ML requires investing in systems that help resolve issues faster. Treating data and ML as real-time products can lead to better value extraction. Service-Level Agreements (SLAs) and reliability benchmarks are becoming more commonplace in the world of data and ML, while troubleshooting will likely become easier over time.

This guide covers how to ingest embedding data and analyze embedding drift for a sentiment classification model using Hugging Face's open source libraries and the Arize platform. The process involves downloading and preprocessing data, training a model, extracting embedding vectors and predictions, logging inferences into the Arize Platform, and preparing data for sending to Arize. The guide also explains how to confirm data is ingested into Arize, track embedding drift, and visualize data using Uniform Manifold Approximation and Projection (UMAP) visualization. By following this guide, teams can monitor their models in production, detect potential performance degradation, and take corrective actions to improve the model's performance.

Arize AI, a machine learning observability platform, has secured $38 million in Series B funding led by TCV with participation from existing investors Battery Ventures, Foundation Capital, and Swift Ventures. The investment marks the largest-ever for an ML observability platform. Founded in 2020, Arize aims to make AI work effectively and fairly. As machine learning models become more complex and widespread, detecting and troubleshooting issues becomes harder. Arize's ML observability platform helps streamline performance monitoring, drift detection, data quality checks, and model validation. The company has also introduced new features such as embedding analysis, embedding drift monitoring, and bias tracing to address these challenges. Arize plans to expand its team across various departments and is currently recruiting for roles in engineering, product, marketing, and sales.