October 2024 Summaries

This post discusses using reinforcement learning and human feedback (RLHF) to improve the performance of a Large Language Model (LLM) on predicting the upvote count of Hacker News (HN) stories. The author, Kyle Corbitt, founder of OpenPipe, explains how they built a reward model that can predict the upvote count based on the story title, URL, date, and content. The model is trained using a dataset of 114K HN stories with their corresponding upvote counts, and the training process takes around 1.5 hours on an H100 GPU for $4.05. The model achieves a root mean-square error (RMSE) of 1.11, which translates to an accuracy of e^1.11 ≈ 3. The author then runs the model against the entire corpus of HN stories and finds that it consistently over-estimates the score at the low end and under-estimates it at the high end. Despite this, the model identifies some great HN stories and provides interesting insights into what makes a story successful on HN. The author concludes by saying that RLHF gives them a powerful set of techniques to improve post quality, which they will cover in the next post in the series.

Fine-tuning is the process of updating model weights to control its behavior, offering deeper and more nuanced control compared to prompting. However, it requires careful consideration as it can lead to "catastrophic forgetting" if not done correctly. Fine-tuning addresses weaknesses in modern frontier AI models, including unreliable adherence to instructions, high costs for operations at scale, and latency issues. By fine-tuning, one can achieve a good dataset, smaller model sizes leading to lower inference costs, and lower latency, making it an attractive option for improving the quality and reliability of GenAI-powered features while reducing costs. Fine-tuning is not a panacea with no tradeoffs, as it requires preparing data, training models, evaluating performance, and deploying the fine-tuned model, which can be achieved by reasonably competent software engineers without specific training in data science or machine learning.

OpenPipe has introduced Direct Preference Optimization (DPO) support, allowing users to align models with their specific requirements more strongly. DPO is an advanced fine-tuning method that enables models to learn directly from preference data, making it useful when users have a source of preference data that they can exploit. This technique is particularly effective when used in conjunction with user-defined criteria and has shown promising results in initial tests, such as reducing the number of responses exceeding word limits by 77% or dropping hallucinated information by 76%. To get started with DPO on OpenPipe, users need to prepare their preference data, upload it to the platform, select the DPO option during fine-tuning job configuration, and launch their training run. The company is also working on integrating DPO into an online learning workflow to enable continual learning.