Using Raft to centralize cluster configuration in Redpanda

Redpanda has introduced a new central configuration system to simplify the management of cluster configurations and minimize operational issues caused by local configuration file discrepancies. Instead of storing configuration in local text files, Redpanda's central system stores uniform cluster configuration properties internally, updating them via an API and replicating them across nodes using Raft, which eliminates the risk of inconsistent configurations. The rpk CLI tool facilitates editing these configurations in real-time, ensuring settings are applied consistently across the cluster without needing node restarts, unless specified. This centralized approach offers a single authoritative configuration source, enhancing reliability and preventing non-deterministic behaviors from node-specific configurations. The system is designed for seamless upgrades, automatically activating for existing users after all nodes are updated to Redpanda 22.1.x, and supporting live configuration changes to most properties. Additionally, it maintains compatibility with Apache Kafka by allowing certain properties to be modified using Kafka's API, although using rpk is recommended for a more robust DevOps experience.