August 2023 Summaries

The NeurIPS 2023 LLM Efficiency Challenge invited participants to produce the strongest language model by fine-tuning a permitted base model for up to 24 hours on a single Nvidia A100 or RTX 4090 GPU, with separate tracks for each hardware type and evaluation based on a subset of Stanford’s HELM benchmark. The guide outlines an end-to-end workflow using Lit-GPT, the organizers’ suggested starter repository, including creating a Python environment, installing PyTorch and dependencies, downloading and converting base-model checkpoints such as Llama 2 or OpenLLaMA, preparing approved datasets, and testing generation. It presents LoRA as a practical baseline for adapting a 7B model within limited GPU memory, while discussing adjustments to batch sizes, sequence length, learning rates, LoRA targets, gradient accumulation, quantization, and mixed precision to balance speed, memory, and model quality. Although Alpaca-Libre is used in examples, the text notes that organizers later prohibited it because the competition disallowed LLM-generated data. Participants are encouraged to evaluate models locally with EleutherAI’s Language Model Evaluation Harness, avoid spending their limited three submissions prematurely, and package final training and evaluation workflows as Docker images according to the official submission repository.