Hunting memory spikes in the Erlang BEAM

New Relic has extensively used Elixir and Erlang's BEAM VM for its high-throughput services, particularly in the New Relic Unified API team, where they focus on understanding both the strengths and potential failure scenarios of these tools. Through rigorous load testing and capacity analysis, they identified memory spikes during requests to downstream services, often caused by a large number of HTTP requests, which sometimes led to out-of-memory (OOM) conditions. Investigating further, they utilized the Erlang Observer to monitor memory usage and discovered that redundant memory allocation in SSL connection processes was the issue, exacerbated by binary DER blobs used for SSL certificates. By switching to an ETS table for certificate storage, they significantly reduced memory usage by 60-70% during expensive queries and improved the system's overall efficiency. This exploration enhanced their proficiency in diagnosing and resolving lower-level issues in Elixir and Erlang applications, reinforcing New Relic's commitment to performance and resilience.