November 2025 Summaries

The text provides a comprehensive tutorial on creating a unified multi-cloud cost dashboard that integrates cost data from various cloud service providers such as AWS and GCP, combined with revenue data from platforms like Stripe, using tools like dlt, SQL, Python, and Rill. It explains the step-by-step process of setting up cost exports from these cloud services, storing the data in DuckDB or ClickHouse, and using Rill for visualization, aiming to provide businesses with a clear overview of their cloud expenses and revenues to calculate margins. The project is designed to be extendable, allowing for the integration of additional data sources and cloud providers, and emphasizes the importance of having a single pane of glass for cost visibility across multiple platforms. The tutorial also highlights the challenges encountered during implementation, such as the manual setup required for AWS and GCP cost exports, but ultimately showcases the value of having a consolidated view of cloud costs and revenues, which can be crucial for financial decision-making in multi-cloud environments.

In a discussion between Joe Reis and Michael Driscoll, the focus shifts from Joe's earlier predictions of intelligence-driven organizations to the current reality of "The Great Reset" in data engineering, prompted by AI's transformative impact. This reset challenges the foundations of the Modern Data Stack era, pushing data teams to re-evaluate their approach as AI reshapes traditional workflows and toolchains consolidate. The conversation highlights the role of hidden "dark matter data," such as Excel sheets and shadow pipelines, which carry significant influence over AI systems. Despite the proliferation of AI-generated solutions, human craftsmanship remains crucial for defining clear and reliable data models. With the rise of autonomous agentic systems, the nature of data team roles is changing, emphasizing intent over execution. Joe expresses optimism about the future, encouraging teams to actively engage with these changes to help shape the next decade in data practices.