Infinite Retention with OpenTelemetry and Honeycomb

Honeycomb is a tool designed for software developers to quickly address production issues by providing fast query responses, even if it sacrifices a slight degree of data completeness. In contrast, compliance and audit workloads demand 100% data completeness over extended periods, often at the expense of query performance. Honeycomb retains telemetry data for 60 days, which is generally sufficient for most organizations, but some require indefinite retention. This text explores an experimental method to achieve infinite retention using Amazon S3, Amazon Athena, and AWS Glue, which allows storing raw telemetry data and querying it over long periods. While this approach offers a solution for long-term data retention and compliance, it results in slower data querying compared to Honeycomb, which is optimized for urgent and iterative use cases. The method involves using OpenTelemetry to export data to an S3 bucket, setting up Glue to structure the data, and using Athena to query it, albeit with a more complex and less user-friendly experience.