Better to Be Wrong Than Vague: Apache Kafka and Software Architecture Predictions for 2021

In a discussion on the future of Apache Kafka and software architecture, experts Gwen Shapira, Michael Noll, and Ben Stopford share predictions about the evolution of Kafka and streaming technologies. They foresee significant enhancements in Kafka's capabilities, including the possibility of running clusters with 10 million partitions and the ability to rapidly scale clusters due to architectural changes like KIP-405 (Tiered Storage) and KIP-500 (ZooKeeper removal). These changes are expected to improve scalability, cost efficiency, and performance, enabling quicker autoscaling and better data management. Additionally, the panel anticipates the widespread adoption of streaming features across various products, suggesting a shift towards an "events first" mindset in software architecture. This paradigm shift could lead to multi-paradigm products that incorporate event streaming into traditional database systems, despite challenges in retrofitting older technologies. The conversation underscores the growing importance of event streaming as a dominant paradigm in contemporary software architecture, driven by evolving user expectations and technological advancements in the Kafka ecosystem.