June 2024 Summaries

Sandy Ryza, Lead Engineer of Dagster Labs, is interviewed on the podcast "Value Driven Data Science" by host Dr. Genevieve Hayes. They discuss the role of data science in data orchestration and how it has evolved with recent advancements in AI and ML. Sandy shares his background in data science, which has influenced his engineering approach to building tools like Dagster that improve understanding and simplify tasks. The conversation highlights the importance of data assets within data science and compares asset-oriented workflows to traditional task-based workflows. The podcast aims to learn more about the intersection of data science and data orchestration, providing a platform for Sandy's insights and experiences.

The rise of medium code is a new class of software practitioners who build and ship mission-critical code to production, but don't necessarily have full-stack engineering skills. These practitioners use more humane interfaces and tools to write code that is precise, testable, and has internal abstractions. Medium code allows more people to write production code more productively, and it has porous borders with traditional software development, making it easier for software engineers to integrate their work into medium-code projects. AI will accelerate this trend across multiple domains, not eliminate it, by increasing productivity and accessibility, but requiring human oversight and interpretation of output. Medium code is the ideal substrate for AI code generation due to its coarse-grained container for business logic, capturing intent and producing discrete artifacts that can be understood by domain experts. The future of medium code is accelerating with AI-native software development, enabling more people to build and ship software, making them economically valuable, and creating a powerful way to work.

In this article, the author shows how to set up a Singer ELT pipeline using Dagster. The setup involves creating reusable resources for the Tap and Target, defining assets that invoke these resources, configuring code location definitions, and setting up a simple job. The author highlights several advantages of running Singer Taps within Dagster, including making Singer an integral part of scheduled pipelines, adding metadata to taps and targets, retrying on failure, sending tap payload to multiple targets, and debugging without hitting the upstream API. However, it also notes that running Singer can increase latency and storage requirements. The author provides a step-by-step guide for setting up this pipeline, including code examples in Python. By leveraging Dagster's orchestration system, developers can create more integrated, observable, and trackable data pipelines using Singer.

Dagster is a powerful data platform that provides various options for implementing Extract, Load, Transform (ELT) processes. ELT has become crucial in moving and transforming data from various sources into centralized data warehouses or lakes. The text discusses why teams might want to look beyond commercial SaaS solutions for data ingestion, such as Fivetran and Airbyte, and explores the different ELT options available with Dagster, including rolling your own solution, using official integrations with Fivetran and Airbyte, and leveraging embedded ELT. Each option has its pros and cons, and choosing the right one depends on specific needs, resources, budget, and goals. Dagster's flexibility and orchestration capabilities make it a powerful solution for managing data workflows, regardless of the ELT approach chosen.