January 2025 Summaries

This article discusses various AI reference architectures, including Prompt Engineering, Retrieval Augmented Generation (RAG), Fine-Tuning, and Pretraining. Prompt Engineering involves designing input prompts to guide the behavior of a Large Language Model (LLM) and is characterized by its ease of use, minimal infrastructure requirements, and ability to develop and experiment at low cost. RAG enhances performance by integrating external knowledge through a retrieval system that fetches relevant data from an external storage layer, allowing for dynamic updates, domain-specific contextualization, and cost efficiency. Fine-Tuning involves using an existing pre-trained LLM and adapting it based on a small dataset for a specific task, offering customization, cost efficiency, and output control. Pretraining a model involves training it on a large volume of data without using any prior weights from an existing model, allowing for avoiding biases, unique data, and different languages. The article also introduces Dagster as a tool that can be used to implement these architectures, providing an example of how to use it for fine-tuning a model.

Traditional software development relies on deterministic processes, whereas AI systems are probabilistic and deal with probabilities, patterns, and distributions rather than fixed rules. This shift in approach requires thinking about confidence levels and anticipating uncertainty when designing AI applications. The right tool for the job is crucial, as different functions suit themselves well to specific AI applications. Design, software architecture, and anything new should be left to people, while agents can autonomously execute complex tasks that traditionally require human oversight. Dagster's abstractions make building production-grade AI implementations more manageable by providing robust frameworks for ETL processes, model training, inference, versioning, testing, and local development. The platform's emphasis on lineage and graph-based thinking helps teams focus on outputs while maintaining visibility into the entire data pipeline.