Reverse Engineering a $500M Mystery: From HashHop to Memory-Augmented Language Models

In 2024, a mysterious AI startup named Magic, backed by significant funding, claimed to develop a groundbreaking model with an extensive context window of 100 million tokens, far surpassing existing models in efficiency and cost-effectiveness. Despite their claims, Magic never released a product, leaving behind only a blog post and a benchmark called HashHop, designed to test the limitations of long-context evaluations through complex token associations. Through reverse engineering, a team achieved perfect HashHop accuracy by treating hash strings as single tokens, demonstrating that the problem of matching arbitrary-length strings can be simplified into key-value lookups via attention mechanisms. This insight led to the development of Memory-Augmented Language Models (MALM), which showed potential in practical applications like code retrieval, exhibiting high accuracy on exact name queries. The work suggests that while Magic's true implementation remains unknown, their approach to tokenization and retrieval could be the real innovation, paving the way for efficient and scalable solutions in AI.