Full Text Search in Pinecone, Now in Public Preview

Pinecone now offers Full Text Search (FTS) in Public Preview, enhancing its semantic search capabilities by integrating keyword matching as a complementary approach. This development allows for both dense and sparse vector searches within a single index, supporting multi-language tokenization and BM25 scoring across multiple fields, which enables precise retrieval of specific information, such as product SKUs or legal citations. The system supports exact phrase matching, present token filtering, and metadata filters, which refine search results before vector ranking, thus improving the retrieval of documents that are semantically relevant and meet specific keyword criteria. The schema for each index is set at creation, and the service supports document operations like upsert, fetch, and delete, with the flexibility to combine scores across different modalities by issuing separate queries. Documentation and examples, such as a Google Colab notebook and a Bird Search demo, are available to guide users through implementing FTS with Pinecone.