June 2022 Summaries

Chris Murphy, Senior Director and Data Scientist at Homepoint, discusses how AI plays an important role in their mission of supporting successful homeownership. He shares his background in physics and transition into financial services, emphasizing the transferable skills in modeling techniques and approaching challenges. Homepoint's machine learning use cases span various areas of the business, including operations optimization, predicting refinancing or delinquency rates, outlier detection, text reading, optical character recognition (OCR), and infrastructure building. To ensure success when applying state-of-the-art ML techniques into established processes, Homepoint's data science team focuses on setting up the right processes from the beginning and maintaining constant communication with business partners and operations teams. They use a variety of explainability and bias tracing techniques to ensure fairness across the board.

The modern machine learning (ML) pipeline relies heavily on big data, with applications such as Mobileye's self-driving car efforts processing over 200 petabytes of data or tens of billions of inferences per day. Kafka is a widely used pub/sub framework that powers event-driven pipelines, offering benefits like asynchronous processing, scalability, and reliability. To monitor ML models, Kafka messages can be ingested into the Arize platform using a simple consumer that consumes micro-batches of events, deserializes them, batches them together, and publishes them to Arize for real-time observation. Arize is built to scale, providing easy ways to ingest data, including Kafka event streams, and unlocking ML performance tracing once ground truths are received, which enables improving model performance by understanding the why and how of models.

Observability is crucial for modern software systems as they can be complex and prone to errors due to billions of lines of code and the integration of data and machine learning models, making it impossible to guarantee flawless performance. Automating observability allows for timely and actionable information about a system's performance, similar to NASA's mission control during the Apollo missions. Different components of modern software systems, including infrastructure, data, and machine learning models, require unique approaches to observability.

Stefano Goria, Co-Founder and CTO of Thymia, discusses the company's mission to improve mental health assessments through a combination of video games based on neuropsychology, facial microexpression analysis, and speech pattern examination. Founded in 2020 amidst the growing global mental health crisis exacerbated by COVID-19, Thymia aims to provide clinicians with objective measures for diagnosing mental health issues. Goria brings his expertise in theoretical physics and machine learning from previous roles at Citi and J.P. Morgan to develop AI systems that underpin Thymia's end-to-end solution. The company focuses on addressing the subjectivity of mental health care by improving assessment quality, particularly for symptoms relevant to depression diagnosis. Thymia collects data from three main sources: video, speech, and behavioral patterns during gameplay. These diverse data types push AI models to their limits, requiring a combination of techniques such as deep neural nets, feature engineering, unsupervised learning, and reinforcement learning. The company also emphasizes ethical use of technology, informed data usage, and transparency in communication with patients.

The company Arize has released a beta version of its embedding drift monitoring and analysis product, which is designed to help machine learning teams troubleshoot models and data that contain unstructured data. The product addresses key challenges such as lack of visibility into what's happening to the data when it's put into production, expensive model training, and difficulty in identifying new patterns emerging from unstructured data. With this release, teams can log models with both structured and unstructured data to Arize for monitoring, enabling them to proactively identify drift and troubleshoot issues using interactive visualizations. The product aims to provide actionable insights to help ML teams improve their models and data, and is designed to work with a wide range of deep learning models and architectures.

Unity Software recently revealed that it missed top line expectations due to issues related to machine learning models, causing an estimated impact of approximately $110 million in 2022. This highlights the growing need for companies to better manage AI risk from both organizational and technical perspectives. There are four steps enterprises can take to prevent common issues with ML before they materially impact revenue: (1) know what can go wrong, (2) implement ML observability, (3) invest in the right people, and (4) ensure ML teams are close to the businesses they serve. By adopting these best practices, companies can better manage AI risk and avoid potential pitfalls.

Embeddings are compact representations of high-dimensional data that help in explaining complex relationships between different inputs. They play a crucial role in machine learning, particularly in reducing the dimensionality of input features and facilitating collaboration across teams. Despite their potential to simplify data, embeddings can still be challenging to understand without additional techniques like UMAP.

Arize, an ML observability platform, has introduced its Trust Center and Security Periodic Table as part of its commitment to robust security, compliance, and privacy. The company recently achieved SOC 2 Type II certification and is pursuing other major industry certifications such as HIPAA. The Arize Trust Center provides an interactive resource for customers and partners to understand the company's governance, policies, and security measures. Security at Arize is built on three pillars: auditability, prevention, and preparedness. These principles are inspired by sectors with effective risk management practices, such as the airline industry. The security periodic table offers an interactive overview of each element of security, along with compliance objectives and overlapping certifications and standards.