Retrieval optimizer: Custom data

Robert Shelton's blog post introduces how to handle custom data schemas within a retrieval optimizer by defining a custom corpus_processor function and tailored search methods. It explains the process of transforming raw data for efficient indexing in Redis and demonstrates how to implement search techniques that utilize specific fields in a dataset, using car manuals as an example. The dataset differs from previous examples, incorporating text chunks and embeddings alongside query metadata, necessitating customized queries to leverage this additional information. The post also discusses the use of query relevance judgments (qrels) to evaluate retrieval performance, showing how custom search methods, including default vector and hybrid search, can be employed. By setting up a search method map and a study config, users can test retrieval methods effectively, with the example indicating improved performance through the use of custom query_metadata fields.