September 2024 Summaries

AI engineering is built on the same fundamental principles as data engineering, with a focus on scalability, reliability, and performance. AI systems rely on high-quality data to build intelligent models, making data quality crucial for effective models. To be successful, AI engineers must adopt data engineering best practices, such as ensuring pipelines are idempotent and repeatable, using scheduling to automate pipeline runs, making pipelines observable, using flexible tools and languages for data ingestion and processing, and testing pipelines across environments before production. The modern orchestration platform Dagster enables AI teams to follow these best practices, providing features like asset-based APIs, declarative automation, observability, and integrations with other environments. By applying these principles, AI engineers can build high-quality, scalable, and reliable AI systems that succeed in real-world scenarios.

Dagster and Modal are used together to automate and streamline machine learning workflows, providing scalable and flexible infrastructure for heavy computing tasks like model training and data processing. Dagster's orchestration features help manage complex configurations, while Modal offers scalability without complexity, allowing teams to focus on development. The tools can be combined to automate podcast summarization, demonstrating their strength in handling parallelism and large-scale data processing. By using Dagster and Modal, teams can simplify pipeline orchestration, reduce infrastructure complexity, and focus on building applications instead of managing complex infrastructure.

The rise of the data platform engineer is a response to the changing landscape of data work, where the initial expectation of Data Scientists doing all the heavy lifting has given way to a more balanced approach. With the decline of the Data Scientist's role and the emergence of new tools like cloud data warehouses and SaaS providers, the focus has shifted towards building platforms that enable Data Scientists to do their jobs effectively. The modern data stack has enabled productivity gains, but pressure is now building on Data Engineers to take on more responsibility. In response, some companies are moving towards a framework approach to data platforms, where Data Engineers build bespoke custom pipelines for downstream consumers without relying on complex higher languages, allowing them to focus on software engineering and career progression. This shift gives Data Engineers something to look forward to, as they move away from just building ETL pipelines and towards more strategic roles.

Dagster is a new paradigm in data orchestration that takes a radically different approach to data orchestration than other tools. It was designed from the ground up with data assets and the full development lifecycle in mind for a more complete and integrated approach to data pipelines. Dagster addresses Airflow's limitations, such as asset orientation, principled architecture, full data engineering lifecycle management, local development and testing, debugging, data lineage and asset management, scalability and isolation, containerization, and CI/CD. It provides better visibility into data lineage and dependencies, rich structured logs for debugging, a local development environment for iteration and error detection, scalable execution environments, isolated environments for tasks, native support for containerized environments, built-in CI/CD practices, automated testing and deployment, and advanced features like data asset management, operational workflows, security and compliance, priority support, context-rich view of assets, platform integration with external assets, searchability and discoverability. Dagster+ adds more features to compete with the best in DataOps, including a built-in data catalog, operational workflows, security and compliance, priority support, context-rich view of assets, platform integration with external assets, and searchability and discoverability. Organizations can use Dagster alongside Airflow or migrate from Airflow to Dagster for better data operations.

The Sakila Co. project is an open-source analytics starter platform built using Dagster, Postgres, dlt, dbt, Clickhouse, and Metabase. The project was created by Pipat Methavanitpong to provide a cost-effective and easy-to-operate data platform for businesses. It uses a combination of best-of-breed open-source solutions to design, run, and observe the entire data platform. The project is fully containerized and can be easily cloned and reused by another company. Pipat chose these technologies due to their high performance and accessibility for most data teams. Dagster was selected over Airflow because it looks at things from a data's perspective, whereas Airflow uses an 'if-this-then-that' approach. The project also includes the use of dlt for incremental sync mode and schema evolution, Clickhouse as an AWS Redshift alternative, and Metabase for user-friendly data visualization. Pipat hopes that this project will help companies start their data-driven journey regardless of budget.

Data visibility refers to the ability to see, track, and understand data as it moves through an organization's systems. It consists of three main components: lineage, monitoring, and access transparency. Lineage is tracing the origin, transformations, and journey of data throughout its lifecycle, while monitoring involves real-time tracking of data health, quality, and performance. Access transparency provides visibility into who is accessing data and how it is being used. Poor data visibility can lead to multimillion-dollar mistakes, reputation-shattering incidents, and legal implications, including operational risks, flawed decision-making, loss of stakeholder trust, financial impact, compliance issues, reputational damage, and business disruption. To avoid these consequences, organizations need to take a proactive approach to data visibility by implementing best practices such as building a solid data platform with access controls in place, using tools that provide real-time monitoring and quality assurance, and ensuring collaboration and governance through a single control plane and metadata management. A reliable partner like Dagster can help organizations achieve better data visibility and reliability, accuracy, and compliance.

The modern data stack has improved upon previous tools, but it also introduces new problems such as fragmented observability, limited orchestration, and high cost. To solve these problems, a new role called the data platform engineer has emerged to manage complex data infrastructure. A good data platform should be scalable and maintainable, have high-quality governance and data observability and insights. It should also support software development lifecycle integration, heterogeneous use cases, declarative workflows, and different languages and tools. Code-based solutions are preferred over no-code or low-code for complex data engineering tasks, and Dagster can help build such a unified data platform with features like code locations and asset checks for data quality and governance. The role of the data platform engineer involves managing complex data infrastructure, building platforms that serve the needs of stakeholders, and moving from individual pipelines to frameworks and services that support the entire data ecosystem within an organization. Building a real data platform is more than just using modern data stack tools, it's thinking about scalability, governance, and observability. Dagster addresses these needs with features that tackle the challenges of today's data engineering.

Mejuri, a global fine jewelry brand, has built an eCommerce data platform leveraging Dagster to manage its vast amount of data from various sources. The company's data team, led by Gabriela, oversees the current-state platform, underlying infrastructure, and DevOps setup, as well as building future features and pipelines. Mejuri uses a "black-silver-gold" structure on Google BigQuery, with dbt Cloud running transformations and Dagster orchestrating all pipelines. The team has successfully transitioned from Airflow to Dagster and appreciates its features, such as pipeline visibility and alerting, which enable smoother development and deployment processes. As Mejuri continues to grow, the data team is looking to empower Analytics Engineers to work independently and improve their CI/CD process.