How we optimized our Akka application using Datadog’s Continuous Profiler

Performance bottlenecks are often unexpected and difficult to locate without proper tools. In this case, a Java application using the Akka framework experienced high CPU usage due to an issue with the ForkJoinPool. The problem was discovered and fixed by utilizing Datadog Continuous Profiler, which measures software resource usage. By analyzing flame graphs and stack frames, it was determined that the LatencyReportActor was responsible for increased time spent inside the ForkJoinPool.scan()/Unsafe.park() methods. Moving this actor to a different dispatcher resolved the issue, reducing CPU usage by 30%. The experience highlights the importance of continuous profiling and not relying solely on intuition when addressing performance issues in production environments.