Orca: Progressive Learning from Complex Explanation Traces of GPT-4 Paper Reading

Orca is a 13-billion parameter model that learns to imitate the reasoning process of large foundation models (LFMs) like GPT-4, surpassing state-of-the-art models by over 100% in complex zero-shot reasoning benchmarks. The paper addresses challenges faced by smaller models such as limited imitation signals, homogeneous training data, and lack of rigorous evaluation. Orca leverages rich signals from GPT-4 to enhance model capabilities and skills through learning from step-by-step explanations generated by humans or advanced AI models.