Monitoring Apache Spark applications running on Amazon EMR

In this guest post by Priya Matpadi, Principal Engineer at Lookout, we learn about implementing monitoring for a Spark streaming application running on Amazon EMR using Datadog. The author explains why they chose to use Datadog and how they integrated it with EMR. They also provide scripts for installing the Datadog Agent on EMR nodes and configuring the Spark check. Finally, the author discusses monitoring custom application metrics in Datadog and creating a comprehensive dashboard for visibility into the health of their Spark streaming application.