July 2022 Summaries

Liftoff is a mobile app optimization platform for marketing and monetizing apps at scale. The company's main mission is to help mobile apps grow and monetize. Machine Learning Engineer Yunshi Zhao discusses her role in training models, deploying them in production, and monitoring their performance. She emphasizes the importance of scalability in data processing and highlights some challenges faced by Liftoff's system built for a specific use case. The company is currently investigating more standardized tools to improve flexibility and applicability across various ML applications. Zhao shares best practices for model experimentation, training, and deployment, as well as the importance of monitoring models in production and addressing feedback loops in ad tech space. She also discusses her involvement in Liftoff's diversity, equity, and inclusion (DEI) committee, focusing on representation in engineering.

Arize AI has hired Claire Longo as its new Customer Success Lead. Longo, a data scientist by training, has experience leading ML engineering and data science teams at Opendoor and Twilio. In her role at Arize, she will focus on ensuring customer success through hands-on support or educational resources. She believes that machine learning observability is crucial for maintaining the quality of models in production and preventing issues from reaching end users. Longo has also developed a library of metrics for recommendation systems to help measure personalization performance.

Three Pitfalls To Avoid With Embeddings` Embeddings are not static and require monitoring to ensure they continue to be meaningful over time. This includes tracking the loss of meaning as new concepts emerge in the real world, which can lead to a non-trivial problem. Monitoring involves setting up a point of comparison with the initial trained embedding and tracking metrics such as average distance between cluster centroids. Proper versioning of embeddings is essential to avoid heartache during iteration on code, while graphing techniques can provide an understanding of how well the embedding performs. Once in production, appropriate monitoring techniques are necessary to ensure consistent value for customers.

Arize AI's Senior Solutions Architect, Suresh Vadakath, brings over a decade of experience in consulting and technical client-facing roles from companies like Dataiku, DataRobot, and Alteryx. He focuses on presenting Arize's platform and ideating on ML observability examples and integrations into the customer's environment. Suresh emphasizes the importance of ML observability for risk management purposes and ensuring timely resolution of issues in high value use cases with growing prediction volumes and large model portfolios. He highlights that a good ML observability platform should provide insights for scenarios involving structured and unstructured data elements in a centralized place. Suresh has observed unique challenges facing financial services companies deploying models into production, such as skepticism from users, data appropriateness and preparation, and regulatory scrutiny on fairness and bias. He believes that communication is key when working with ML observability systems due to the constant gray areas involved.

Dimension reduction techniques are crucial in data science for visualization and pre-processing in machine learning. Three popular dimensionality reduction techniques are SNE (Stochastic Neighbor Embedding), t-SNE (t-distributed Stochastic Neighbor Embedding), and UMAP (Uniform Manifold Approximation and Projection). These neighbor graph algorithms follow a similar process, starting with computing high-dimensional probabilities p, then low-dimensional probabilities q. The cost function C(p,q) is calculated by comparing the differences between probabilities, which is then minimized to obtain human-interpretable information from the embedding space.

From Physicist to Machine Learning Engineer Justin Chen transitioned from academia to machine learning after receiving his PhD in Physics from Rice University. He initially struggled to showcase his skills to the industry, but shifted his focus to highlighting mathematical methods and coding work. Chen emphasizes the importance of being able to communicate code with others as a key skill for success in machine learning engineering. He worked on various projects at Manifold AI, including end-to-end ML pipelines, and developed expertise in handling sensitive data in the healthcare space. Chen advises starting with basic models, focusing on relevant metrics, and identifying subject matter experts to narrow down feature sets. In his current role at Google, he focuses on speech recognition and audio processing, using techniques like NLP and explainability to address challenges. Chen stresses the importance of monitoring models regularly, having performance metrics, and having a human in the loop to detect bias and improve fairness. He also notes that working for a startup can provide excitement and flexibility, but may not offer the same level of resources or growth opportunities as a bigger company.

Khyati Sundaram, CEO and Chairperson of Applied, is on a mission to improve diversity in hiring by leveraging lessons from behavioral science and technology. The company's platform aims to provide unbiased hiring solutions, focusing on skills-based testing and decision intelligence systems. By removing noise from resumes and using machine learning to optimize the hiring funnel, Applied seeks to empower humans to make fairer decisions at scale. However, Sundaram acknowledges that biases can occur in various stages of the ML model lifecycle and emphasizes the need for full testing in real-world environments. To address this, Applied is working on optimizing human judgment and machine learning data, aiming to create a platform where everyone cares deeply about quality matches and knows what high ROI looks like. Ultimately, Sundaram's goal is to build a society-wide expression of inclusivity through her company's innovative solutions.

Arize:Observe Unstructured, the first summit dedicated to unstructured data initiatives, recently concluded with nearly 500 technical leaders and practitioners in attendance. The event highlighted four key takeaways: monitoring of unstructured data has arrived; what works in training may not work in production when deciding what to label next; the rise of single, unified models will change MLOps; and cutting-edge machine learning is becoming more accessible. Arize's support for embedding analysis and drift monitoring is now available as part of its free subscription tier, enabling teams to log models with both structured and unstructured data for monitoring purposes.

Malav Shah, a Data Scientist II at DIRECTV, has an interesting career journey in machine learning (ML). He initially studied information technology but developed an interest in AI during his undergraduate years. After completing his Master's degree in Computer Science with a specialization in ML from Georgia Tech, he joined AT&T and later moved to DIRECTV. At DIRECTV, Malav applies modern ML techniques to deliver innovative entertainment experiences. The company's ML organization is structured as a center of excellence responsible for solving problems and developing solutions for stakeholders while defining the infrastructure that these teams will use. Key areas of focus include content intelligence, recommendation engines, computer vision, natural language processing (NLP), and monitoring model performance in real-time to address concept drift issues. Malav advises new data scientists to focus on understanding the underlying data and business impact rather than obsessing over perfect metric scores right away. He also highlights the evolving MLOps and ML infrastructure space as an exciting era for machine learning innovation.

Arize has appointed Matt Wilson as its new Head of Sales, bringing a decade of experience working with large enterprises and establishing product-led growth motions to the role. Wilson joins Arize from Pendo, where he was an early sales hire and most recently served as RVP of Enterprise Sales, helping the company achieve over $100 million in revenue and a valuation of $2.6 billion last year. As Head of Sales at Arize, Wilson aims to accelerate and grow the closed pipeline, increasing revenue through acquisition, retention, and net renewals, while also hiring and growing a world-class sales organization. Wilson believes that machine learning observability is crucial for businesses, having seen firsthand the impact it can have on productivity and financial losses. He also emphasizes the importance of leading with positive intent, creating a culture of collaboration, and being open to feedback and ideas from his team. With experience scaling Pendo and now joining Arize, Wilson sees parallels between the two companies in terms of their focus on product-led growth and delivering value through their products.