The Query Tantivy Couldn't Save in OpenObserve: 2.6s to 89ms for Random High-Cardinality Lookups

The text discusses performance optimization for high-cardinality random ID lookups in OpenObserve, focusing on the limitations of previous improvements that reduced query times significantly for selective, range-prunable queries. Specifically, it addresses the challenge posed by queries involving random high-cardinality lookups, such as those using trace_id, where traditional range-pruning techniques are ineffective. To overcome this, the text introduces a transposed bloom filter layer designed to minimize S3 round trips by restructuring data layout, enabling a single read for multiple file checks. The bloom filter's effectiveness is highlighted through its ability to significantly reduce query times for random IDs, though it is less beneficial for time-ordered IDs where tantivy's sparse index is already efficient. Operationally, the bloom filter requires careful enablement based on field characteristics, and it is intended as an optional feature to be selectively applied to achieve optimal performance improvements.