The Bitterest Lesson

Machine learning progress is often attributed to increasing compute, as described by Rich Sutton’s “bitter lesson,” but the passage argues that task selection is even more important, followed by data, compute, and algorithms. While games make compute appear especially decisive because their objectives are clear and training data can be generated through self-play, real-world applications require people to define objectives that genuinely reflect user and system needs. Models can achieve strong loss curves and technical performance while remaining unhelpful if they optimize the wrong goal. The author argues that ML research often prioritizes algorithms and scaling over the messier work of collecting appropriate data and understanding the surrounding product, organization, or users. OpenAI’s InstructGPT work illustrates this point: comparatively small GPT-2-scale models trained to follow instructions outperformed much larger GPT-3 models trained only for next-token prediction, showing that aligning a model with the right task can matter more than vast increases in scale.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.