June 2026 Summaries

Lumi Loglake and Splunk's SmartStore both facilitate the querying of log data stored in object storage like AWS S3, but they differ significantly in their architecture and operational models. While SmartStore requires data to be copied from object storage to a local indexer cache for querying, Loglake allows queries to be executed directly in the object storage using virtual compute pools that activate only when needed. This architectural distinction impacts performance, scalability, and cost, with Loglake offering a more flexible and efficient solution by eliminating the need for constant infrastructure and enabling workload isolation. SmartStore, although innovative in 2018 for reducing retention costs, suffers from limitations like cache thrashing and fixed infrastructure requirements that can degrade performance during extensive searches. In contrast, Loglake's approach of running queries where data resides, without data movement or proprietary constraints, allows for seamless integration with existing Splunk dashboards and tools while also supporting broader query access through SPL, Spark SQL, LogQL, and ANSI SQL.

Machine data architectures are evolving as organizations increasingly move logs and machine data to object stores like AWS S3, driven by growing telemetry volumes and rising costs. This evolution is facilitated by modern lakehouse platforms separating storage and compute, which allow retention of more data at lower costs compared to traditional observability/SIEM infrastructures. A significant challenge with this architecture is the need for structured logs and pre-defined schemas, which can hinder rapid, exploratory investigations required during security incidents. Unlike traditional schema-on-write pipelines that demand predefined schemas and structured data, Lumi's Loglake feature leverages schema-on-read, enabling direct querying of unstructured logs in object storage. This approach allows teams to dynamically reconstruct context and optimize performance without the delay of schema definitions, making operational and security investigations faster and more efficient. As the industry shifts towards separating storage and compute, the focus is on enabling instant searchability and understanding of large volumes of machine data without unnecessary preprocessing, which Lumi and Loglake aim to address.

As organizations increasingly store log data in AWS S3 for cost efficiency and extended retention, tools like Lumi Loglake and Splunk Federated Search for S3 offer different methods for querying this data. Lumi Loglake allows teams to use standard SPL to query logs stored in S3, returning native Splunk events that seamlessly integrate with existing dashboards, alerts, and workflows, without requiring schema definitions or AWS Glue Catalogs. This approach supports querying unstructured logs directly, making it suitable for dynamic environments where log formats and fields may change frequently. In contrast, Splunk Federated Search for S3 uses a separate query model based on the sdselect command, returning results as tables, which may necessitate modifications to existing dashboards and alerts to maintain workflows. This method requires initial schema and metadata setup through an AWS Glue Data Catalog, aligning with structured data practices but potentially imposing limitations on flexibility. Thus, while both solutions address cost-effective log querying from S3, they differ significantly in terms of integration, data preparation requirements, and the ability to handle unstructured data.

Organizations are increasingly storing log data in AWS S3 to reduce expenses and extend retention, and both Lumi Loglake and Splunk Federated Search offer solutions for querying this data. Lumi Loglake allows users to query logs in S3 using standard SPL, producing native Splunk events that integrate seamlessly with existing dashboards and alerts, providing a familiar experience for analysts. In contrast, Splunk Federated Search requires the use of a separate query model based on the sdselect command, resulting in table-based outputs that may necessitate adaptation of existing workflows. Additionally, Loglake supports querying unstructured logs without prior schema definitions or catalog maintenance, enabling immediate search capabilities, while Splunk Federated Search requires schema representation through an AWS Glue Data Catalog. This fundamental difference highlights Loglake's flexibility in handling dynamic log data, positioning it as a more integrated solution for teams already using Splunk's infrastructure.