August 2024 Summaries

In this post, we explore preference tuning of LLMs through a practical case study on summarization, using Ray and Anyscale as our compute platform. We applied Direct Preference Optimization (DPO) to the Mistral-7B-Instruct-v0.1 model to produce good summaries for CNN articles. Our results show that DPO is effective in tackling specific domains such as summarization where there is no ground-truth response, and it can achieve much higher win-rates than using supervised fine-tuning or prompting GPT-4o. We also found that both β and learning rate are critical for performance and may require a thorough hyperparameter search. Additionally, we demonstrated the effectiveness of regenerating preference training data with the new model and applying additional rounds of DPO to achieve even more gains in performance.

Ray Training Day is a full-day intensive training event designed to enhance AI and ML skills in GenAI and LLMs. The event offers sessions for both beginners and advanced practitioners, covering topics from the fundamentals of Ray to advanced techniques for building scalable AI systems. By attending, participants will gain practical knowledge and insights into simplifying large-scale machine learning workflows, making it easier to handle complex AI projects. The training is led by experts in the field, including the original creators of Ray, and covers a range of topics including RAG applications, LLMs, and MLOps best practices. With five major learning paths to choose from, participants can select sessions that suit their needs and goals, and receive hands-on training on building and deploying scalable AI systems. The event is only available in-person in San Francisco on September 30, 2024, and tickets are expected to sell out quickly.