January 2024 Summaries

Axiom's Bring Your Own Bucket (BYOB) option allows customers to use their private AWS S3 buckets for data storage while maintaining the same features, UI, and performance as Axiom's standard SaaS configuration. This approach caters to organizations with stringent compliance requirements, enabling them to manage data governance and migration to the cloud at their own pace while retaining full ownership and control of their event data. Customers benefit from zero maintenance obligations, unrestricted data access, and the ability to utilize their event history for AI and machine learning applications without incuring additional costs. Axiom handles all software, server maintenance, and upgrades, providing a cost-effective solution for enterprises that prefer to keep their data within their cloud accounts, ensuring portability and accessibility even if they choose to discontinue Axiom services.

SaaS products have emerged as an alternative to traditional on-premise software, offering benefits such as reduced issues with licensing and piracy, yet they face challenges related to data custody and retention, particularly in regulated industries like healthcare. Axiom, a company focused on capturing and querying organizational data, has introduced the "Bring Your Own Bucket" (BYOB) feature, allowing enterprise customers to use their own AWS S3 storage for data, thereby maintaining ownership and control while leveraging Axiom's computing power. This approach addresses data custody concerns and compliance requirements, offering a more understandable and streamlined solution for data storage and processing. Axiom's design philosophy emphasizes clarity and simplicity, ensuring that data management is intuitive and compliant with industry standards like HIPAA and SOC2. The company has also optimized its architecture by removing unnecessary components to enhance performance and ease of integration for enterprise customers.

Tomás Senart, a principal engineer at Axiom, shares his holiday-driven experimentation with optimizing the performance and efficiency of Axiom's columnar database, which is designed to be cost-efficient and time-focused. Axiom allows for variable schema databases where events can have different fields and types, which enhances ease of data ingestion and integration. The architecture partitions data into blocks for storage and querying, using a custom Decomposition Storage Model (DSM) format that optimizes compression. Senart's recent work involved significant contributions to bitmap serialization, reducing costs and latency while maintaining performance. He experimented with dictionary encoding for _time columns, discovering that using dictionary encoding and frame of reference encoding can significantly reduce memory usage and improve performance. Although challenges remain, such as dealing with datasets with wide timestamp ranges, Senart remains optimistic about future improvements, driven by empirical validation and a passion for making systems faster.

Axiom BYOB is a new option designed for Enterprise plan customers who wish to use Axiom's services while keeping their event data stored in their own controlled environment. This feature allows customers to replace Axiom's S3 storage with their own, enabling Axiom's cloud-based ingest and query functions to read and write data securely to the customer's space. BYOB offers the same interface and query language as the standard Axiom service, ensuring a seamless user experience with no additional maintenance required. Customers maintain complete control over their data, with the ability to extract raw format event data using provided binaries, ensuring data ownership and longevity. The service promises the same performance and reliability as when Axiom manages the datastore, while offering the added advantages of data access control and retention flexibility.

Landon Lewis discusses how to enhance data analysis using Axiom by integrating open-source threat intelligence, which can transform volumetric data sources into valuable insights without compromising on data retention and storage. By utilizing Vector as a syslog aggregator, users can manage syslog payloads effectively, even when they deviate from standard protocols. The process involves defining enrichment tables using CSV files created via Python scripts, which contain data types such as IP, Domain, URL, Hash, and Email. The Vector Remap Language (VRL) is then employed to filter log messages, extract relevant fields, and compare them against the enrichment data to identify potential threats, adding enriched threat information to the logs when matches are found. This method offers the advantage of real-time data enrichment, better aligning with contemporary detection engineering efforts.