Open Sourcing Vespa, Yahoo’s Big Data Processing and Serving Engine

Vespa, Yahoo's open-source big data processing and serving engine, has been released to enable developers to build applications capable of real-time computations over large datasets at internet scale. This initiative follows Yahoo's tradition of open-sourcing big data infrastructure, aiming to address the challenges of serving results quickly, as required in applications like search, recommendation, and personalization. Vespa is designed to handle complex computations at the time of serving rather than pre-computing, which facilitates a better user experience and reduces hardware costs. The engine is already in use across various Yahoo brands, efficiently processing billions of daily requests with low latency. Vespa distributes data and computations across multiple machines, ensuring no single point of failure, and it is flexible enough to be run on-premises or in the cloud, with Docker images and rpm packages available for easy deployment. The release is intended to empower developers to build feature-rich applications without the need to manage complex distributed systems, with comprehensive documentation provided to get started quickly.