October 2022 Summaries

Dagster is a data orchestrator that helps build and operate machine learning pipelines. It models graphs of data assets and the data transformations that connect them, allowing users to define training and batch inference pipelines in Python and run them reliably in production. Unlike workflow managers like Apache Airflow, Dagster is designed for use during development and modeling, with a lightweight execution model that enables quick iteration and easy debugging. It also models data assets, not just tasks, which helps users understand how changes to one asset affect the entire pipeline. By using Dagster, machine learning engineers can try out ideas faster, translate those ideas into production more easily, and understand how they're performing over time.

The Human Genome Project successfully mapped the human genome in April 2003, but only completed 85% of the genome mapping until January 2022. Genetic data is challenging to work with due to its large size and complexity. Zephyr AI, a Washington DC-based company, uses predictive analytics to sift through patient data and find patterns or "interrelationships." The team specializes in ingesting large amounts of complex, disparate molecular, experimental, and clinical data using novel machine learning algorithms to draw new insights in the field of patient care, drug development, and healthcare administration. Zephyr AI has adopted Dagster, a software-defined asset framework that provides a declarative approach to managing and orchestrating data. The company uses Dagster to build pipelines with powerful abstractions, a single pane of glass for observability, and a centralized control plane with shared logging and governance. With Dagster's help, Zephyr AI has streamlined its development process, accelerated speed, and improved transparency in their work, allowing them to maintain one master orchestration view while managing multiple distinct deployments with isolated codebases. The company plans to incorporate model training pipelines written in R and expand their models to bring better outcomes and a better healthcare system to more people.

DuckDB is a powerful, feature-rich SQL engine that can run locally and efficiently access remote data sets. It's gaining popularity due to its ease of use, speed, and flexibility. However, it has limitations, such as being designed for single-machine use and not suitable for large-scale data processing. To overcome these limitations, DuckDB can be combined with other technologies like Dagster, S3, and Parquet to create a powerful multiplayer data lake. The author of the article is building a project called "DuckPond" that uses DuckDB to process data from Wikipedia and stores it in a Parquet file on S3. The project also includes tests and an I/O manager to handle input/output operations. Despite its potential, DuckDB is not yet ready for widespread adoption, but it could become a popular choice for subsets of workloads that don't require ultra-high scale.

A data pipeline smoke test is a technique used to speed up data pipeline development by automatically running all data transformations on empty or synthetic data, exercising every transformation inside the pipeline. This approach catches bugs in just a few seconds and can significantly reduce development time. It is particularly useful for pipelines with heavy business logic and can be integrated with various frameworks such as Pandas, SQL, Spark, or Dask. The test verifies that code in each transformation follows the rules of the data processing language and that each transformation can handle the type of data produced by upstream transformations. Data pipeline smoke tests can also help avoid accidentally breaking pipelines in production and provide broad test coverage.

Marco Salazar, the author of this text, is a developer working on Dagster, an open-source tool for building and running data pipelines. Recently, their cloud product was launched, leading to increased usage by new and existing customers, which resulted in some new and exciting problems. The log viewer, a critical component of the application, wasn't loading for one customer due to a massive influx of log lines per second. This issue led Marco and his team to investigate how the log viewer works and triage the problem. They discovered that the client CPU was pegged at 100% due to parsing and rendering competing for the main thread. To solve this, they implemented web workers, which significantly reduced CPU utilization. However, upon further investigation, they found that using a `cursor` variable in the subscription hook caused issues with subscription tearing down and re-creation. Solving these problems involved restructuring queries to not include the `cursor` variable and moving their use of `useSubscription()` into a component that returned null. The root cause fixes were deployed without introducing web workers, resulting in improved performance and user satisfaction. The team learned valuable lessons about profiling, intuition, solving for one problem causing another, and using external resources like Google and Stack Overflow.

Dagster is a data orchestration framework designed to serve developers at every stage of the software development lifecycle. The five stages of the software development lifecycle are writing code, testing, review and collaboration, deployment, and monitoring and observability. Dagster provides support for all these stages through its declarative programming model, software-defined assets, built-in testability, branch deployments, local development, config-as-code, and declarative scheduling features. These features help developers write high-quality code efficiently and effectively, test their code thoroughly, review and collaborate on code, deploy their code successfully, and monitor the performance of their software. Dagster is designed to serve every stage of the development lifecycle, unlike existing solutions like Airflow and Prefect that have limitations when it comes to serving all stages.

Dagster is a data orchestrator that helps build data pipelines, similar to how Django is used for building web applications. It enables users to produce data assets and then re-run these processes incrementally when the asset becomes outdated. Dagster has three key differentiators: local development and testing, software-defined assets (SDAs), and decoupling pipelines from the environment. These features make it easier for data engineers and machine learning engineers to build complex data applications. The tool provides a framework for building data pipelines, including tools for ingesting data, combining and transforming it, storing it in a place where it can be used, and re-running these processes when necessary. Dagster is designed for users who need to bridge the engineering and data worlds, providing a platform for managing complex data applications.

Postgres, a relational database management system, was chosen over a message queue like Kafka for Dagster Cloud's logging system due to its familiarity and performance. Despite initial concerns about using a relational database as a message queue, Postgres proved suitable with proper configuration, such as archiving and rate limiting, which helped scale the database and mitigate issues like unbounded growth and spikes in customer log volume. The decision was also influenced by the philosophy of choosing the right tool for the job, rather than relying on new infrastructure, and minimizing additional costs associated with learning a new technology. While Postgres has its limitations, such as not supporting multiple regions or requiring downtime during failovers, the author acknowledges that these issues can be addressed in future work.