Pre-training a 1.11B LLM on a 6 GB Laptop GPU — Measured, Not Claimed

A community project reports combining block-coordinate optimization, CPU weight offloading, tied embeddings, gradient checkpointing, chunked cross-entropy, and ternary-weight training techniques to pretrain a 1.11-billion-parameter language model on an RTX 4050 laptop GPU with 6 GB of VRAM, using roughly 4.5–4.9 GB in tested configurations. The author measured throughput of up to 1,579 tokens per second at a 4,096-token context length, but estimates that Chinchilla-style training on roughly 33–36 billion tokens would still require about 375 days, emphasizing that memory reductions do not remove the compute constraint. The model uses a published gated DeltaNet-style recurrent architecture supplemented by two sliding-window attention layers, which improved short-range exact-token recall in synthetic tests while retaining constant-memory recurrent inference characteristics. The work argues that the stack raises the measured 6 GB pretraining capacity from about 243 million parameters with a standard script to roughly 1.58 billion, while projections for larger GPUs are explicitly labeled as extrapolations rather than measurements. It also details RAM requirements, Windows VRAM-spill risks, software pitfalls, rejected optimization attempts, retracted claims, and limitations including that models above 48 million parameters were not trained to convergence and several results rely on single-seed experiments.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.