January 2019 Summaries

Vespa, an open-source big data processing and serving engine primarily developed by Yahoo engineers and used by platforms like Yahoo News and Yahoo Finance, has released several updates in January 2019 to enhance its functionality. These updates include the introduction of support for multiple levels of parent-child document references, allowing fields to be imported with minimal performance impact and simplifying updates in many-to-many relationship use cases like Product Search. Additionally, Vespa now supports file URL references in application packages, enabling the inclusion of large data files without storing them directly in the package, as they are automatically downloaded during redeployment. A new SyncFeedClient API for Java is also introduced to facilitate high-performance batch data feeding from systems without full streaming support, such as Kafka and DynamoDB. The Vespa team encourages community feedback and contributions to further improve these features.

Vespa introduces a cost-effective approach to personal search at a massive scale through a method called streaming search, which eliminates the need for maintaining expensive global indexes by utilizing separate small indexes per user. This approach significantly reduces the cost associated with index updates and queries, which is a major issue in traditional methods for handling large personal data stores like Gmail. Vespa's streaming search mode operates by storing raw user data in a log-level store, distributing data across nodes for efficient query handling, and implementing a full search engine over the raw data without global indexing. The solution offers benefits such as reduced costs and stable latencies over time, making it suitable for applications like email search, personal suggestions, and private content searches. Vespa's implementation allows for structured and text search, advanced relevance, and features like faceting, providing a scalable and proven framework for personal search solutions.