Full Text Search in SmithDB: Designing an Inverted Index for Object Storage

SmithDB is designed to support full-text search and JSON filtering over agent traces with a median latency of 400 ms, despite dealing with large, deeply nested JSON documents stored in object storage. Unlike traditional search engines like Lucene or Tantivy, SmithDB approaches indexing from first principles due to the unique challenges posed by the large payloads of agent traces, which are mainly composed of extensive inputs and outputs that far exceed traditional metadata columns. This requires a compact, efficient inverted index capable of handling the power-law distribution of term frequencies and supporting multiple query modalities. SmithDB's architecture leverages object storage to keep compute stateless, which allows for scalable operations by merely adding nodes. It employs a specialized storage layout for its inverted index, organized by byte-budgeted row groups, to optimize query performance by minimizing object storage requests and efficiently managing term frequencies. The system is engineered to integrate with Apache DataFusion and Vortex, ensuring that search operations can be pushed down through the same scan pipeline as other queries. The design also accommodates the need for compact storage and efficient query execution by using FST-based term dictionaries and block-bitpacked deltas for postings and positions, ensuring high performance even with large, complex data sets.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.