How we built high speed threat hunting for email security

Sublime is an email security platform designed to enable rapid historical threat hunting and detection backtesting, crucial for security teams such as SOC, incident response, and detection engineers. Utilizing Message Query Language (MQL), Sublime efficiently processes millions of messages by separating data into warm and cold storage, querying lightweight metadata first to minimize cold storage requests, and parallelizing operations for speed. The platform employs a two-phase approach comprising candidate selection and evaluation, where cheap operations filter potential messages before expensive operations further refine the selection. Innovative techniques like predicate pushdown and period chunking enhance query performance, while a robust caching strategy optimizes enrichment functions like ml.logo_detect. Sublime addresses challenges like database dead tuples and high DB load by leveraging indexed cursors and aggregation tactics, ensuring scalable and swift hunts at enterprise levels. Future improvements aim to dynamically adjust processor allocation based on demand and utilize lossy data structures to enhance candidate selection, maintaining the platform's commitment to speed and efficiency in threat detection.