pg_textsearch 1.0: How We Built a BM25 Search Engine on Postgres Pages

Todd J. Green's blog post discusses the development and implementation of a native BM25 search engine for Postgres, known as pg_textsearch, which is now available to Tiger Cloud customers and as open-source software. Unlike previous solutions that required additional systems like Elasticsearch for full-text search, pg_textsearch integrates BM25 ranking directly into Postgres, improving search performance and ranking quality. This is achieved through features like Block-Max WAND optimization for efficient top-k queries, SIMD-accelerated decoding for reduced index size, and a parallel index build process. The extension is designed to overcome limitations of Postgres's built-in ts_rank, such as lack of inverse document frequency and term frequency saturation, by implementing a robust architecture that includes a hybrid memtable and segment design inspired by LSM-trees. Benchmark tests on the MS-MARCO dataset demonstrate significant speed and efficiency improvements over existing solutions like ParadeDB, especially for shorter queries. Despite these advancements, the current version of pg_textsearch has limitations, including lack of support for phrase queries and fuzzy matching, with future updates planned to address these areas and enhance overall functionality.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.