Using ElasticSearch and Logstash to Serve Billions of Searchable Events for Customers

Rackspace's Mailgun has effectively implemented Elasticsearch and Logstash to manage and analyze billions of email events monthly, enhancing customer access to detailed event data. By utilizing Elasticsearch's real-time indexing and robust querying capabilities, Mailgun overcame limitations of its previous log system, enabling full-text searches and event filtering through a new API and control panel. Events are stored in a distributed, fault-tolerant cluster setup, allowing for efficient data retrieval and management, while Logstash aids in collecting and processing these events. Despite Elasticsearch lacking built-in authorization features, Mailgun developed a two-stage proxy to secure access, and it uses daily indices for efficient data retention. The integration of Kibana further aids in visualizing and making sense of the logged data, providing users with a significantly improved experience in managing and analyzing their email events.