March 2024 Summaries

The alignment of generative models with human feedback has notably enhanced the performance of natural language generation tasks, with methods such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) proving more effective than supervised fine-tuning (SFT) alone. These approaches aim to align large language models (LLMs) to produce outputs that match human preferences, thereby preventing the generation of illegal or incorrect content. RLHF, for instance, involves a cycle of data collection, reward modeling, policy optimization, and iterative refinement to align AI behavior with human values. DPO, on the other hand, focuses on adjusting policies based on preference without explicit reward modeling, while methods like Kahneman-Tversky Optimization (KTO) incorporate human psychological biases into the learning process. Additionally, Self-Play Fine-Tuning (SPIN) leverages synthetic data to enhance LLMs without relying on extensive human-annotated data. These methods, coupled with ongoing developments such as ORPO, aim to improve the reliability and usefulness of AI models in line with human expectations.

This blog post provides a comprehensive guide on deploying Large Language Models (LLMs) serverlessly using Modal Labs, focusing on Mistral-7B-instruct by Mistral AI. It outlines the process of serverless deployment, emphasizing the cost-effectiveness of this approach as charges are based on computational usage rather than fixed resources, and highlights the potential drawback of cold starts when servers reactivate after idling. The tutorial explains how to set up and configure a serverless deployment using Modal's Python interface, detailing the creation of necessary files like constants.py for configurations, engine.py for the inference engine, and server.py for the REST endpoint. The post elaborates on using GPU configurations for model deployment, leveraging a Docker container environment, and implementing Modal stubs for efficient resource management. It concludes by describing the deployment process using the Modal CLI, suggesting that serverless deployment is ideal for varying usage patterns and promising future discussions on alternative serverless providers like Beam Cloud and Runpod.

Recent developments in the field of language models have focused on the potential of small language models (SLMs) such as TinyLlama, Phi2, Gemma, and StableLM2, which may offer comparable performance to larger models while being more manageable and efficient. This exploration involves fine-tuning SLMs to perform specific tasks with the same efficacy as larger models, leveraging distributed training techniques such as Distributed Data Parallelism (DDP) and Fully Sharded Data Parallelism (FSDP) to manage the challenges posed by training on massive datasets. The integration of Ray, a tool for distributed computing, plays a crucial role in managing distributed computing tasks, ensuring fault tolerance, and optimizing resource utilization. The training infrastructure is supported by a cluster of nodes with shared volumes for data streaming, using WebDataset for efficient data handling. Despite encountering challenges such as data loading and streaming interruptions, the setup aims to demonstrate the viability of SLMs in real-world applications, with future releases planned to showcase fine-tuned models for specific tasks.