How to Send Data through Logstash or Kafka from Elastic APM

In modern deployments where application servers often have short lifespans or use unreliable hardware, Elastic APM Server, starting from version 6.4.0, can send data to more reliable systems like Logstash or Kafka, thereby offloading reliability requirements. This setup allows for data enrichment through Logstash plugins and involves configuring the APM Server to output events to Logstash or Kafka, which then forwards them to Elasticsearch. Each event type, such as transactions or spans, generates a daily index, and specific configurations are necessary to ensure the correct handling of APM data. Source maps, essential for mapping obfuscated code to original sources, require special consideration and should ideally be sent directly to Elasticsearch. Kafka can be introduced to buffer events before they reach Elasticsearch, with configurations allowing for topic-based organization of data. The integration of these systems enables more robust and flexible handling of application performance monitoring data, with the Elastic APM team encouraging feedback and contributions from users.