August 2018 Summaries

Comet.ml's Project Visualizations tool enhances the ability of data science teams to efficiently compare and analyze numerous machine learning model iterations through advanced visualizations, fostering a more iterative, collaborative, and reproducible environment. By automatically tracking datasets, code changes, experimentation history, and models, Comet.ml provides data scientists and machine learning engineers the means to identify the most effective models—known as champion models—by offering insights into the hyperparameter sets and model configurations that yield the highest accuracy. The visualization options include line charts for tracking training loss, bar charts for identifying top-performing models, and parallel coordinates charts for exploring hyperparameter spaces. These tools not only allow for efficient comparison and exploration of experiments but also facilitate sharing results and generating insights, ultimately supporting faster iteration cycles and robust model development.

The Comet.ml team continues their participation in the Kaggle Home Credit Default Competition, focusing on creating a comprehensive dataset for their models by incorporating features from seven different datasets, with insights from mortgage professionals. This involves manual feature engineering, assisted by subject matter experts, to identify key features indicative of an applicant's default risk. The manual feature engineering led to a slight improvement in model performance, with an AUC score increase from 0.745 to 0.7519. The team connected with industry professionals Cody Dadiw and Philippa Stewart-Donnelly, who provided valuable insights into feature selection while highlighting the importance of human oversight in machine learning applications within the financial sector. This collaboration underscores the potential benefits of combining human expertise with machine learning for better decision-making, emphasizing the need for ethical considerations and compliance with financial regulations. The blog also reflects on the limitations of data-driven models in capturing the nuanced human aspects of lending, advocating for hybrid approaches that integrate human judgment with AI.

The latest episode of the In Context podcast, hosted by Kathryn Hume, features a discussion with Eric Colson, Chief Algorithms Officer at Stitch Fix, exploring the interplay between machine learning, data science, and business operations. While the episode title suggests a focus on hiring for autonomy, the conversation covers various aspects of structuring data science teams, the role of algorithms in the customer experience at Stitch Fix, and the balance between machine learning and human judgment. Colson elaborates on how Stitch Fix leverages algorithms beyond just recommendations, integrating them into inventory and demand management, and highlights the company's organizational structure that aligns data science closely with other business functions. He advocates for the concept of "full stack data scientists" to enhance the iterative process and reduce coordination costs, contrasting it with specialized roles seen in other companies. The episode also touches on how data scientists can contribute to the business by understanding the broader "Horizon of Imagination" and the importance of a quantitative mindset in candidates. The conversation raises questions about the feasibility and benefits of adopting a generalist approach in data science teams and how such a model could be implemented in different company cultures.

The article explores the concept of cross-validation in machine learning, emphasizing its role in evaluating model performance and reducing bias compared to other methods like simple train/test splits. It focuses on the k-fold cross-validation variant, where the dataset is divided into K partitions, and the model is trained on K-1 partitions while tested on the remaining one, iteratively evaluating and averaging the test errors for accuracy assessment. Despite the advantage of producing less biased performance estimates, a notable downside is the increased training time since the model is trained K times. The article provides a practical example using the Scikit-learn library and the KFold class, demonstrating how to implement k-fold cross-validation for a text classifier and highlighting the importance of not using the test set until the experimentation is complete to avoid overfitting. Additionally, it introduces comet.ml, a platform for tracking machine learning experiments, founded by Gideon Mendels.

Comet.ml has introduced a feature called the Query Builder to enhance the discoverability and organization of machine learning experiments by allowing users to filter experiments based on metrics, metadata, and parameters. This feature addresses user feedback regarding the need for better experiment management, enabling users to perform complex filtering to identify top-performing models and save these filters as Saved Queries for future use. Additionally, the Query Builder can be combined with a Group By feature to further organize experiments, such as by learning rates. Comet.ml aims to improve experiment transparency and collaboration, akin to how GitHub transformed code management, by offering a platform that tracks datasets, code changes, and experimentation history. The company is led by CEO and co-founder Gideon Mendels, who has a background in NLP and worked at Columbia University and Google.