November 2024 Summaries

The Dagster + Sigma integration bridges the gap between business intelligence and data orchestration, providing a unified view of data pipelines from raw sources to BI dashboards. This integration enables data teams to track dependencies and lineage across the entire data lifecycle, fostering collaboration and improving operational efficiency. With this integration, data engineers can create end-to-end pipelines that cover the entire data journey, while analysts can be confident that their reports are always based on the latest data. The integration also supports scalability, centralized data management, and automated updates to ensure business intelligence remains accurate and timely without manual intervention.

Dagster has introduced new integrations with popular business intelligence tools—Looker, Power BI, Tableau, and Sigma—to enhance the visibility and management of data flows from source to dashboard. These integrations allow BI assets to be directly represented within the Dagster asset graph, improving understanding of data lineage and aiding in the debugging of data quality issues. By triggering compute processes in BI tools for cached assets, the need for scheduled processes is reduced, facilitating seamless data updates and distribution. This makes Dagster a comprehensive platform for managing analytics environments, providing insights into both data warehouses and dependent dashboards and reports. Future improvements aim to enhance user experience, expand integrations, and offer toolkits for custom integrations, further unifying data management and visualization efforts.

Analytiks, a company founded by Mathieu Stark, has developed an innovative approach to help small and medium-sized businesses harness the power of modern data orchestration using Dagster's commercial platform. The company's pre-packaged solution, called the "DAD stack," makes Dagster+'s enterprise features accessible to organizations of all sizes. Analytiks offers a unique deployment model that allows clients to access Dagster+ features at a fraction of the cost, with a focus on providing comprehensive support and ongoing maintenance. By leveraging Dagster+'s advanced capabilities, Analytiks enables its clients to build robust data pipelines, integrate complex data models, and unlock significant business value through automated reporting and real-time decision-making. The company's approach has proven particularly valuable for organizations dealing with complex data challenges, and it plays a crucial role in expanding Dagster's ecosystem by making its tools accessible to a broader range of organizations.

Servco, a Hawaiʻi-based company founded in 1919, has transformed its data management system from disconnected legacy systems to a unified cloud-native platform using Google Cloud's BigQuery and Dagster. The new approach allows for self-service culture where analysts can access well-documented, trusted data directly, improving efficiency and enhancing data usage across the organization. Key takeaways include choosing the right orchestrator, building semantic models, unified tools for efficiency, and preparing for governance in data workflows. Servco's transformation is ongoing, with plans to migrate more data sources to the cloud and scale their models as user access increases. The company aims to share data by demystifying it and making it accessible in a usable way, enabling internal customers to see value in accessing the data themselves.