January 2025 Summaries

NVIDIA's A10, A100, and H100 GPUs offer varying performance-to-cost ratios, making them suitable for different machine learning tasks. The H100 is ideal for large language model workloads with high precision requirements, while the A100 is a versatile GPU suitable for larger models with moderate precision needs. In contrast, the A10 and L4 GPUs are more cost-effective options for smaller models or inference tasks. When selecting a GPU, consider factors such as task type, model size, memory requirements, budget, and performance needs to choose the best fit for your specific use case.

The MTEB leaderboard is a comprehensive benchmark that evaluates the performance of embedding models across various tasks, providing a standardized way to compare different models. While high ranking on the leaderboard doesn't guarantee the best fit for a specific use case, considering factors such as task-specific performance, computational requirements, and domain relevance can help make an informed decision. Top models currently on the MTEB leaderboard include generalist embedding models like NV-Embed-v2, Nomic-Embed-Text-v1.5, and bge-en-icl, which have been fine-tuned for specific tasks or domains such as medicine, finance, law, code, math, Japanese, Korean, Chinese, French, Arabic, among others. Domain-specific embedding models can offer superior performance for specialized applications, making it essential to explore these models alongside top performers on the leaderboard to find the best fit for a particular use case.

Axolotl is a wrapper for lower-level Hugging Face libraries that simplifies the fine-tuning process of large language models, offering granular control while being easier to use. It comes with built-in default values and optimizations, including sample packing, which can improve training efficiency. Axolotl allows users to train open weights models like LLaMA 3/LLaMA 3.1, Pythia, and Falcon on their own data without needing to implement the fine-tuning process from scratch. Unsloth is a framework designed to dramatically improve the speed and efficiency of LLM fine-tuning, allowing users to fine-tune Llama 3.1, Mistral, Phi & Gemma LLMs up to 2-5x faster with 80% less memory usage compared to FA2. Torchtune is a PyTorch-native library for easily fine-tuning LLMs, offering a lean and extensible design that's just pure PyTorch, with excellent interoperability with popular libraries across the PyTorch ecosystem. The choice between these tools ultimately depends on specific requirements, hardware constraints, and level of expertise, with Axolotl being recommended as a good starting point for beginners.

As a cloud compute platform, Modal is committed to customer security and privacy as top priorities. Our product is secure by design, and we take measures from development to deployment to mitigate risk and earn the trust of our users. We have achieved SOC 2 Type I compliance last year and announced support for HIPAA compliant workloads earlier this year. Recently, we completed a more rigorous SOC 2 Type II audit with no deviations found, demonstrating our commitment to continually improving our security posture. As our customer base grows, we will renew our SOC 2 audits annually alongside product and process improvements, providing stronger reassurances around the security of our platform.