Company
Date Published
Author
Jim Hagan
Word count
3042
Language
English
Hacker News points
None

Summary

When working with distributed logging, a developer's first instinct might be to send application logs directly from their application to a logging backend over transactional REST APIs. However, this approach can lead to issues such as backpressure on HTTP requests, latencies causing lags in transmission of log data, and network connectivity problems leading to lost logs. Decoupling the log forwarder from the application code can make the system less fragile by keeping complex processing computations separate and updating configurations without affecting underlying applications. Log forwarders provide built-in memory or file-based buffering, flexibility for latencies and interruptions, and support for various network protocols and output formats. Choosing a suitable log forwarder is crucial, with options like Logstash, Rsyslog, Fluentd, Wayfair's Tremor, and Timber.io's Vector available. These forwarders can be used in co-located or separately located configurations, with load balancing providing scalability and high availability. Enriching, filtering, and routing logs can further improve the log forwarding process by adding or altering content to records, excluding unwanted content, and sending different types of logs to different backends. Ultimately, a well-designed log forwarding system can reduce fragility and ensure observability in distributed systems.