September 2024 Summaries

Nebius has established its first Paris data center at Equinix’s PA10 campus, leveraging sustainability initiatives such as using waste heat to warm an urban farm on the facility's roof. This center, which will exclusively use Nebius-designed servers certified by the NVIDIA-Certified Systems program, represents a strategic move in expanding their network of availability zones globally. The Paris data center will begin handling client workloads in November, with a rapid setup period of just two months, thanks to Nebius's hardware R&D optimizations. This development is part of Nebius’s broader strategy to enhance its AI infrastructure with minimal environmental impact, following a similar sustainable model used in its Finnish data center, which supports over 2,000 households through waste heat. The expansion plans include increasing capacity in Finland, leveraging colocation facilities, and pursuing new greenfield projects, reinforcing Nebius's position as a leading provider of accelerated computing capacity in Europe.

Mikhail Mokrushin, the Managed Schedulers Team Leader at Nebius, introduces "Soperator," a project aimed at integrating Slurm and Kubernetes to optimize distributed model training and high-performance computing (HPC). The initiative addresses the challenge of combining Slurm's efficient scheduling with Kubernetes' autoscaling and self-healing capabilities by representing Slurm clusters using Kubernetes resources. Soperator allows for the creation of Slurm clusters with Kubernetes Pods, maintains a shared root filesystem to simplify node management, and includes GPU health checks for reliable performance. The Kubernetes-first approach employed by Nebius preserves user familiarity while adhering to cloud-native constraints, enabling easy scaling and high availability. Although the project is still evolving, it provides a significant step towards seamless integration of Slurm and Kubernetes, offering a more efficient and user-friendly solution for managing computational resources in large-scale machine learning environments.

Soperator, a Slurm-based workload manager operating within a Kubernetes cluster, is introduced as an open-source solution to enhance the orchestration of large machine learning models in multi-node GPU environments. By integrating the advanced job scheduling and hardware control of Slurm with the scalability and flexibility of Kubernetes, Soperator simplifies scaling and cluster management, making it GPU-ready. This innovation addresses limitations of traditional Slurm setups, such as the necessity for identical cluster nodes, by introducing a shared root file system and a Terraform operator, enhancing user experience and reducing the need for in-house DevOps expertise. The system also features a hardware health check mechanism to ensure fault-tolerant training by monitoring GPU status and reallocating workloads if issues arise. Available on GitHub, the first public version of Soperator is designed to be production-ready and aims to continuously evolve with community and market needs, focusing on improving security, scalability, and support for future software and hardware updates.

Machine learning model inference is the application of a trained model to real-world data, producing actionable outputs for practical use. This process follows a comprehensive development lifecycle, beginning with data collection and preparation, followed by model pre-training, fine-tuning, and culminating in deployment. During the training phase, models learn to recognize intricate patterns in datasets tailored to specific tasks, while large-scale models undergo pre-training for a generalized understanding before fine-tuning for specialized tasks. Once ready, the model is deployed on a high-performance server, handling real-time data streams and making predictions that guide business decisions. The inference architecture integrates data sources, utilizes GPU-enabled hosts for efficient processing, and ensures predictions reach their destinations, such as databases or dashboards. Despite challenges like team collaboration, costly hardware, model drift, scalability, and interpretability, machine learning inference remains crucial for extracting insights and supporting business strategies.

Hadoop and Spark are two prominent open-source technologies used for processing large-scale data in pipelines, each with distinct purposes and strengths. While Hadoop provides a comprehensive framework encompassing data storage and processing via its components like HDFS, MapReduce, and YARN, Spark serves as a more advanced data processing engine that enhances Hadoop's capabilities with faster in-memory computations and streamlined processes through its DAG execution model. Spark offers a unified API for various data processing tasks and integrates seamlessly with machine learning and real-time processing applications, making it highly suitable for modern analytics. Despite Spark's superior processing speed and ease of use, Hadoop is still favored for cost-effective storage and scalability, especially when security and flexibility are paramount. The two technologies often complement each other, with Spark leveraging Hadoop's storage layer for enhanced performance. Managed Spark services further simplify operational complexities, allowing engineers to focus more on developing machine learning applications without dealing with infrastructure challenges.

Automated machine learning (AutoML) revolutionizes the development of machine learning models by automating complex tasks such as data preprocessing, feature engineering, model selection, and hyperparameter tuning, thereby enhancing scalability, efficiency, and accessibility. AutoML platforms simplify the machine learning process by creating multiple training pipelines that test various algorithms and parameters, allowing even non-experts to participate in model development through user-friendly interfaces. AutoML not only speeds up the workflow but also provides more accurate outputs compared to hand-coded algorithms, bridging the skills gap for those with domain knowledge but lacking technical expertise. The approach mitigates the 'black box' problem in AI, making models more transparent and easier to understand. Despite its advantages in efficiency, cost savings, and accessibility, AutoML is not a replacement for human expertise, requiring oversight from data scientists. Its applications span across industries, including finance, healthcare, marketing, and more, promising to democratize machine learning and drive innovation while also taking a step towards artificial general intelligence.

The LMSYS competition on Kaggle challenged participants to predict human preferences when comparing large language models, using tools like the Chatbot Arena and an unbiased Elo rating system to aggregate results. The competition was demanding, requiring substantial computational resources, and participants like the author, who work with large language models (LLMs) daily, found it both instructive and enjoyable. The event highlighted advanced techniques such as datasets, model comparisons, pseudolabeling, LLM ensembling, and optimizations for training and inference. A noteworthy strategy involved model distillation, where a large model was trained and its predictions distilled into smaller models, exemplifying the technique's effectiveness. The winning approach used distillation with a 5-fold setup and averaged LoRA layers to create a single efficient model, demonstrating the ongoing potential of model distillation in improving language model performance.

Features play a crucial role in machine learning, serving as the key attributes of a dataset that enable algorithms to discern data patterns and make accurate predictions. The process of feature engineering involves generating these attributes from raw data using mathematical operations and domain knowledge, significantly impacting model performance. Feature selection further refines the dataset by using statistical and computational methods to identify the most relevant features, enhancing model optimization. Various methods like filter, wrapper, and embedded techniques are employed to assess and select features, each with its advantages depending on the context. In modern applications, advanced techniques such as neural networks facilitate feature learning, which is essential for complex data types like images and audio. This iterative learning process is adaptable, allowing models to keep pace with evolving data and maintain prediction accuracy. However, feature learning also faces challenges such as data quality, computational cost, and interpretability. Overall, features are fundamental to any machine learning project, as they determine the model's ability to map information and achieve optimal performance.

Entropy, a concept borrowed from thermodynamics, is used in machine learning to measure the randomness or disorder within a system, particularly in supervised learning models which analyze pre-labeled datasets to predict new data. Introduced by Claude E. Shannon, entropy quantifies the unpredictability of class label distributions, with higher entropy indicating more randomness and making accurate predictions more challenging. In decision trees, entropy helps in classifying data by minimizing disorder through strategic splits, with information gain representing the reduction in entropy. Entropy is also used beyond classification in areas like dimensionality reduction, anomaly detection, and as a loss function in model evaluation. However, its effectiveness can be compromised in datasets that are noisy, contain errors, or are highly imbalanced, leading to potential biases toward majority classes. In such cases, alternatives like Gini impurity, balanced accuracy, and hinge loss might be considered. Understanding entropy and related concepts is essential for machine learning practitioners, and structured training, such as a machine learning bootcamp, can provide practical insights into applying these principles effectively.

NVIDIA has reduced the hourly cost of its H100 GPU-equipped virtual machines from $4.85 to $3.50, with further discounts available for longer-term reservations. Meanwhile, the new H200 SXM GPUs, now integrated into Nebius, boast significant advancements over the H100, including nearly double the memory capacity and enhanced performance capabilities, such as faster data access speeds and improved inference performance with large language models. The H200 GPUs are available starting at $2.50 per hour. Additionally, Nebius has introduced a managed service for MLflow, a popular tool for streamlining machine learning workflows, now in public preview, making it more accessible to a wider audience. At the AI Summit in London, Nebius' Product Director Narek Tatevosyan discussed the essential steps and challenges involved in developing generative AI models. Their blog offers insights into optimizing machine learning experiments, selecting storage for deep learning, and exploring the similarities between different language model architectures like transformers, RNNs, and SSMs.

ML model development is complex, requiring the coordination of multiple roles and extensive trial and error, often within costly computing environments. MLflow has become a favored tool for many ML teams due to its ability to streamline workflows by providing experiment tracking, model management, and enhanced collaboration. The introduction of Managed Service for MLflow on Nebius offers a cloud-hosted solution that simplifies infrastructure maintenance, allowing teams to focus on model development without worrying about hosting or security issues. This service provides a fully managed environment that enhances reproducibility, facilitates efficient model management through a shared Model Registry, and improves cross-functional collaboration. With Managed Service for MLflow, ML teams can achieve production-ready models more quickly, leveraging the platform's capability to capture metadata and optimize the MLOps lifecycle. The tool's integration with Nebius and other cloud endpoints ensures seamless operation and is currently in public preview, offering a free and user-friendly experience akin to traditional SaaS.