Why the OpenTelemetry Batch Processor is Going Away (Eventually)

The OpenTelemetry community has shifted its recommendation away from using the batch processor in production environments due to its limitations in handling data during Collector restarts and other failure scenarios. Historically, the batch processor buffered telemetry in memory, resulting in data loss if the Collector crashed or restarted because the telemetry was not durably stored before acknowledgment. Instead, the community now recommends exporter-level batching with persistent storage, which enqueues telemetry in a durable queue within the exporter before sending an acknowledgment. This approach aligns with an at-least-once delivery model, enhancing data durability and reliability across restarts. By consolidating batching and queueing within the exporter and writing data to disk early in the pipeline, exporter-level batching addresses the shortcomings of the batch processor and provides coordinated backpressure, reducing CPU overhead and memory pressure, and shifting the tradeoff to managing disk capacity and I/O performance.