How to build a machine data lake
Blog post from Axiom
Axiom presents a machine data lake as a way to retain and query logs, traces, metrics, and security events without splitting them across aging systems or making older data difficult to access. Its design separates event datasets from purpose-built metrics datasets, recommends distinct OpenTelemetry datasets for logs, traces, and metrics, and uses correlation groups to connect related data; security events are stored in an OCSF dataset with documented attributes typed as columns and unfamiliar attributes preserved in an unmapped field. Schemas are derived from incoming data, allowing structured OpenTelemetry records to become queryable at ingestion while query-time parsing supports older unstructured data. Axiom argues that columnar compression makes retaining all events affordable, citing reductions of about 25 times—and up to 50 times for some structured data—rather than relying on sampling. Data remains accessible through one query path as it moves from a write-ahead log into compressed object-storage blocks and automatically shifts among access tiers, while serverless query compute is kept separate from storage. The approach can also coexist with Splunk workflows, enabling teams to add high-volume or previously uncovered security sources before deciding whether to expand.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| OpenTelemetry | 12 | 822 | 142 | 56 | -4% |
| Serverless | 2 | 833 | 236 | 92 | -40% |
| Observability | 1 | 2,390 | 523 | 156 | -23% |
| Platform Engineering | 1 | 1,295 | 290 | 84 | -36% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.