Company
Date Published
Author
Odette Harary
Word count
3795
Language
English
Hacker News points
None

Summary

A Dagster tutorial on creating an ML pipeline for fine-tuning Large Language Models (LLMs) using LoRA and parameter-efficient techniques. The authors share their findings and demonstrate best practices in creating a clean production ML pipeline, including operationalizing and keeping the model up to date, monitoring quality, and automating the pipeline with Dagster's resources and asset-based coding. The tutorial covers topics such as choosing the right LLM, using notebooks to build fine-tuning models, converting notebooks to Dagster code, thinking in assets and resources, tokenizing data, building an ML pipeline, evaluating model performance, and automating the pipeline. The authors provide a comprehensive guide on how to create a production-ready ML pipeline with Dagster, making it easier for machine learning teams to streamline their workflows and improve productivity.