HELM: benchmarking large language models on the Together Research Computer

The Stanford Center for Research on Foundation Models (CRFM) has announced a comprehensive effort to benchmark 30 language models, known as Holistic Evaluation of Language Models (HELM), which covers 42 scenarios from question answering to sentiment analysis. To support this effort, the Together Research Computer aggregates idle GPU cycles across thousands of servers, running inference over more than 11 billion input tokens and 1.6 billion output tokens on 10 open language models, including GPT-3, Flan-T5, BLOOMZ, and Galatica. This decentralized computing approach aims to make the field more accessible to researchers and practitioners by solving computation bottlenecks in large language models. The project is an early step towards enabling efficient, shared compute for AI, with the goal of bringing the world's compute together to enable everyone to contribute to and benefit from advanced AI models.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.