Company
Date Published
Author
Shahar Fleischman • Baruch Brutman • Sonya Liberman
Word count
2172
Language
-
Hacker News points
None

Summary

Outbrain, a leading content discovery platform, transformed its content recommendation system into a scalable search problem using Elasticsearch to handle its vast and complex requirements. The system indexes articles as separate Elasticsearch documents with semantic features to determine relevance, and user interests are translated into Elasticsearch queries, allowing for personalized content recommendations. Market rules, such as geographic targeting, are implemented as filters in Elasticsearch to ensure compliance. To incorporate advanced machine learning models, Outbrain developed custom Elasticsearch plugins, optimizing for high throughput and low latency by separating indexing from querying, utilizing a read-only index, and employing techniques like force merging and efficient data node management. These efforts allowed Outbrain to serve 800,000 requests per minute at under 100 milliseconds latency, enhancing the personalization and relevance of its recommendations while maintaining scalability and efficiency.