December 2025 Summaries

Building a data platform that scales effectively with a business requires proactive architecture rather than reactive measures, as reactive approaches often lead to fragmentation and maintenance challenges. The text discusses the importance of choosing the right orchestrator early in the process, as it serves as the heart of the platform, ensuring consistent standards and reducing onboarding times. It highlights the benefits of a composable data platform with interchangeable components, allowing for flexibility and easier integration of new tools. Case studies from companies like smava, Otto, and Big Cartel demonstrate the advantages of centralized control planes and unified orchestration, reducing downtime and improving observability. Successful platforms enable faster development, reliable data quality, and improved collaboration, while poorly architected ones lead to inefficiencies and trust issues. The text emphasizes the need to consider both technical architecture and team dynamics as the platform grows, offering insights into dividing responsibilities and maintaining clarity and confidence.

In the realm of data engineering, selecting the right data ingestion pattern—push, pull, or poll—is crucial for building reliable and maintainable pipelines, as demonstrated using real Dagster code examples. Ingestion is often an afterthought in data engineering, leading to challenges when source systems change, requiring robust and scalable solutions. Push-based ingestion, initiated by the source system, is effective for real-time data delivery but offers less control over timing and volume. Pull-based ingestion, controlled by the data platform, provides scheduling flexibility but depends on source systems exposing APIs. Polling-based ingestion combines aspects of both, checking for new data frequently, but it involves complex state management. Modern data platforms ingest from a variety of sources, necessitating consistent patterns to avoid issues like inconsistent error handling and data quality problems. This guide emphasizes the importance of idempotency, schema management, observability, and error handling to avoid technical debt and operational headaches. While building custom solutions can be a valuable learning exercise, the guide suggests leveraging managed solutions like Fivetran and open-source ones like Sling to focus on high-value engineering work.

The process of deploying a trained model as a serverless endpoint involves orchestrating the model's transition from raw data to a functional, user-interactive stage using tools like RunPod and Dagster. This guide details how to create a RunPod endpoint that hosts the model, leveraging serverless infrastructure to provide scalable, cost-efficient access. The deployment requires building a Docker image that includes all necessary scripts and dependencies, which is then pushed to a registry. The serverless handler functions as the interface for inference requests, similar to AWS Lambda. By representing the serverless endpoint as a Dagster asset, developers can manage infrastructure creation and track lifecycle, ensuring reliable integration into the pipeline. Additionally, a Dagster asset called chat_inference facilitates interaction with the endpoint, allowing for structured input and output management, which aids in maintaining a record of the model's performance over time. This structured approach not only enhances model deployment but also supports future adjustments and retraining, underscoring the importance of orchestration in modern machine learning workflows.

The integration between Dagster Cloud and Atlan streamlines the synchronization of data pipeline events and asset materializations by enabling real-time streaming of asset events directly into Atlan's data catalog. This integration reduces the need for users to switch contexts when managing data pipelines in Dagster and cataloging assets in Atlan by creating a live connection that automatically syncs asset materializations, failures, run statuses, and lineage relationships, along with rich event metadata. It operates through a dedicated event consumer that monitors the Dagster Cloud event log and forwards relevant events to Atlan, enriching the Atlan catalog with operational metadata and ensuring it reflects the actual state of the data pipelines. The integration is available for all customers of Dagster Cloud and Atlan, and feedback for further improvements is encouraged.

The process of training a large language model (LLM) involves multiple stages coordinated through Dagster, emphasizing reproducibility, scalability, and GPU efficiency. The initial steps involve gathering data, training a tokenizer in Rust, and preparing the training environment by packaging necessary code and dependencies into a Docker image. Training is conducted on GPUs using RunPod, which facilitates resource management without manual intervention, aligning with the structured three-stage nanochat training pipeline: base pretraining, midtraining, and supervised fine-tuning. This setup allows for flexible scaling based on data size and model complexity. The use of Dagster assets enables detailed tracking and versioning of each training step, while real-time monitoring of GPU utilization via RunPod provides insights for performance tuning. After training, the model undergoes validation with academic-style benchmarks to assess its generalization capabilities, though initial runs on minimal resources may result in lower performance. The next steps involve deploying the model using serverless solutions, completing the end-to-end orchestrated pipeline from data ingestion to deployment.

The exploration of nanochat, an educational language model, emphasizes understanding the complexities of language model development and maintaining clarity in the process. Unlike state-of-the-art models, nanochat serves as an educational tool within a single repository, showcasing how these systems are built, focusing on the workflow rather than just the model itself. The project integrates with Dagster for managing data ingestion, tokenization, training, and validation, emphasizing modularity and observability. By using a curated subset of the FineWeb dataset and employing a Rust-based tokenizer for efficiency, the pipeline demonstrates the importance of organized data management and reproducible workflows. Validation is elevated to a prominent role, ensuring data usability before proceeding to modeling stages. This series aims to enhance the training process's visibility and reproducibility, with future installments focusing on modeling workflow and practical training considerations.