Company
Date Published
Author
Maxim Brown
Word count
1382
Language
English
Hacker News points
None

Summary

Maxim Brown discusses the challenges of storing and analyzing large volumes of web access logs from technologies like NGINX, Apache, and AWS Elastic Load Balancing (ELB). He explains that these logs can provide valuable key performance indicators (KPIs) for monitoring application health and user experience. However, storing all log data can be expensive and difficult to maintain, making it essential to find ways to extract meaningful insights from the noise. Maxim Brown introduces the concept of log-based metrics, which involve generating metrics from specific parts of log events to track trends and identify issues without indexing large volumes of logs. He provides best practices for creating these metrics, such as identifying what information in logs is valuable to monitor, using attributes like client IP, requested URL path, and response status code, and converting other attributes into tags to filter and aggregate metrics. Maxim Brown also discusses how log-based metrics can be used with centralized monitoring platforms like Datadog to visualize, alert on, and correlate logs with infrastructure metrics and traces, making it easier to track long-term trends and perform anomaly detection and forecasting.