Flink's 95% problem

Apache Flink, initially adopted by major tech companies like Alibaba, is marketed as an ultra-low latency tool for complex event processing with exactly-once semantics, but it is often seen as overly complex for most real-world applications. While it offers advanced features, its high complexity and operational challenges make it suitable for only about 5% of use cases, such as strict, mission-critical operations seen in companies like Uber. Many streaming needs can be met with simpler solutions like HTTP services paired with Postgres or OLAP databases such as ClickHouse. The engineering burden and steep learning curve associated with Flink have led to limited adoption and financial returns, as evidenced by acquisitions of Flink-related startups by larger companies without substantial business growth. The complexity of integrating Flink into existing systems, due to issues like schema evolution and configuration complexity, makes it an impractical choice for many companies compared to more straightforward, widely-understood technologies like SQL databases. Despite its potential, Flink's intricate nature and the availability of simpler alternatives suggest it is not the general-purpose processing framework for most organizations, highlighting the lesson that technical sophistication does not equate to widespread practical use.