OpenAI's Parameter Golf: Train the Best Language Model That Fits in 16MB on Runpod

Parameter Golf is a competitive challenge by OpenAI aimed at training the most efficient language model within a 16MB artifact and under a 10-minute training window using 8×H100 GPUs on Runpod, with the goal of achieving the lowest bits per byte (BPB) on the FineWeb validation set. The competition emphasizes creativity and efficiency over brute computational power, focusing on techniques such as depth recurrence, parameter tying, quantization-aware training, novel tokenizers, and low-rank factorizations. It runs from March 18 to April 30, 2026, with OpenAI providing $1M in compute credits to assist participants. As a talent pipeline for OpenAI, the challenge functions similarly to competitive mathematics and programming olympiads, allowing strong participants to catch the eye of the research team. The leaderboard has seen rapid progress, with scores improving from a baseline of 1.2244 BPB to 1.1228 BPB in five days, driven by techniques like quantization-aware training and sliding window evaluation. Participants submit through pull requests to the parameter-golf repository, with records requiring statistical significance and a minimum improvement of 0.005 nats over the current state of the art.