How a regex simplification in Loki increased performance by up to 300x

Loki, a log aggregation system developed by Grafana Labs, faced a performance bottleneck due to the slow regex package in Go, which hindered efficient log searching. The Go regex package, while prioritizing security against ReDoS attacks, was not performant for high-volume log filtering in Loki. To address this, the team implemented a workaround by leveraging the syntax sub-package that allows regex simplification, effectively using byte comparisons instead of complex regex operations. This approach significantly improved query performance, reducing execution times from 11.5 seconds to 1.5 seconds for certain queries, with overall speed enhancements ranging from 5x to over 300x. The improvement maintains the original security features of the Go regex package, as it does not replace it but optimizes its use. The team plans to expand this solution into a package for broader use and presented future plans for Loki at GrafanaCONline 2020, including enhancements to the LogQL query language.