The Last Mile is Solved by Slop

Adrian Brudaru describes a new approach for automating complex data modeling processes using a prototype called "slop," which leverages large language models (LLMs) to transform raw data into a dashboard-ready star schema with minimal human intervention. The traditional manual slog of inspecting data, designing layers, and writing SQL is replaced by an automated workflow that bridges the gap between raw data and usable business insights. This is achieved through a scaffold that runs a pipeline, extracts schemas, and guides the LLM to build a star schema, reducing the time required from days to minutes. The process involves anchoring the data with evidence, defining goals through specific questions, and employing a scaffold as a programmable skill to guide the LLM. The prototype demonstrates that with the right structure, an LLM can automate complex tasks like dimensional modeling, commoditizing traditional data engineering skills and shifting the value to defining desired outcomes. The ultimate vision is for the system to achieve full autonomy, where human validation is replaced by automated testing, allowing the agent to self-correct and complete tasks without manual prompts.