Exa AI Research Blog | Semantic Search & Neural Network Search Engine

Building a modern search engine involves managing the complexities of ingesting and querying the ever-changing web, characterized by heterogeneous content, varying update frequencies, and sheer volume. The in-house data processing framework, exa-d, was developed to address these challenges by optimizing typed columns with declarative dependencies, enabling engineers to focus on data relationships rather than update steps. This approach allows for efficient management of data updates, whether through surgical updates or full rebuilds, without unnecessary rewrites, thanks to exa-d's ability to identify affected rows and columns precisely. The framework ensures efficient parallel execution by distributing workloads across heterogeneous resources and minimizes redundant computation by leveraging a storage model that tracks data completeness. Using Ray Data for query planning, exa-d computes only necessary updates, maintaining a dynamic and scalable search index. As the web evolves, exa-d continues to adapt, offering a robust solution for maintaining derived states over an extensive web index.