How data engineering fails

In his Datanova talk, Benn Stancil examines potential failures in data engineering by drawing parallels with the film "World War Z," emphasizing the need to anticipate scenarios where popular tools like Snowflake, Fivetran, and DBT might not succeed as expected. Stancil identifies several challenges, including the monotony of data engineering tasks, the high costs associated with data engineers and tools, and the risk of roles being replaced by automation or decentralized data management trends such as data mesh. He also discusses the possibility of AI automating many data engineering functions, potentially transforming the role into one more focused on infrastructure maintenance. Stancil advocates for a shift from technology-centric to problem-centric approaches, urging the data engineering community to focus on creative, empathetic problem-solving by understanding real-world issues faced by users rather than solely relying on technological solutions. This approach would involve listening to the broader community's needs to successfully navigate the evolving landscape of data engineering in the age of AI.