How Reinforcement Learning from AI Feedback works

In this article, we discuss the RLAIF (Reinforcement Learning from AI Feedback) method for creating harmless and non-evasive language models. This approach was introduced by OpenAI in their paper "Constitutional AI: Harmlessness from AI Feedback" [1]. The main idea is to use reinforcement learning to train a language model based on feedback from another AI system that has been trained on a constitution defining ethical principles. The RLAIF method consists of four steps: pretraining, critique generation, reward modeling, and fine-tuning with PPO (Proximal Policy Optimization). The model is initially pretrained on a large corpus of text data. Then, the critique generation step involves training another AI system to generate critiques based on the constitution principles. These critiques are used to provide feedback in the reward modeling step, where the language model's behavior is assessed according to how well it adheres to the constitution. Finally, the fine-tuning step uses PPO to improve the language model's performance while maintaining its alignment with the ethical principles defined by the constitution. RLAIF has several benefits compared to RLHF (Reinforcement Learning from Human Feedback), including better scalability and potentially lower costs due to using computer labor instead of human labor for generating feedback data. Additionally, RLAIF may offer some ethical improvements by filtering out extreme views during the democratic constitution creation process. In summary, RLAIF is a promising approach for creating harmless and non-evasive language models that align with specific ethical principles defined in a given constitution. Further research into this method could help improve the safety and reliability of AI systems as they become increasingly integrated into our daily lives.