January 2019 Summaries

To analyze my billing data in a useful manner, I wanted to understand how code changes and features impact the cost of a running system. I started by building a serverless application using AWS Lambda for normalizing some of the billing report data which was then deposited into S3. However, as my analysis with AWS Athena progressed, I outgrew it due to the large number of individually stored objects in S3. To add a new feature for improved data analysis capability, I considered running an Elasticsearch cluster or using Amazon OpenSearch service, but opted for CHAOSSEARCH instead. I chose CHAOSSEARCH because of its ease of integration with my existing data in S3 without requiring any additional development work, single source of data truth, scalability, and familiarity with a Kibana-based interface. This decision allowed me to quickly analyze my data and focus on solving the problems inside my data rather than how to analyze it.