Lessons Learned From the Migration to Confluent Kafka

Honeycomb's platform team recently underwent a challenging migration to a new ingest pipeline architecture for customer events, which highlighted the dynamic changes in the performance envelope of critical systems and their operational impacts. The migration process involved updating the Kafka cluster to utilize tiered storage in Confluent Platform 6.0, allowing for significant reductions in storage costs and operational resources by offloading cold Kafka segments to S3. Despite careful planning, the transition faced several setbacks, including missteps in instance type selection, network resource saturation, and unexpected outages, which underscored the importance of planning for operational concerns beyond steady-state scenarios. The experience revealed the need for manual intervention capabilities when automation fails, as well as the necessity of allocating sufficient resources and gradually adjusting operational limits to manage unforeseen pressures. Honeycomb learned valuable lessons about managing performance and operational envelopes, which they are eager to share with the community to enhance collective understanding of best practices in system migrations.