Train a Model on Your Own Agent Sessions

A pipeline built around labeled coding-agent sessions from Paper Compute exports conversations, turns, labels, and training inputs into Databricks Unity Catalog, where they can be inspected and transformed into fine-tuning and evaluation datasets. Labels such as golden, regression, pushback, apology, and no-outcome distinguish reviewed successful sessions, failures, engineer corrections, candidate review points, and unproductive work, enabling traceable selection decisions. Training data consists of eligible or manually approved sessions, while evaluation data includes held-out good sessions and correction-based regression cases, with full sessions kept together to avoid leakage between datasets. Corrections are converted into tests by providing a model only the conversation before the mistaken response and using the later engineer feedback as a separate judging guideline. MLflow tracks datasets, model runs, and comparisons between a base Qwen3-4B coding model and a LoRA-tuned version, using a Databricks-hosted LLM judge to assess whether responses meet recorded expectations. The approach emphasizes reproducible evidence, separation of training from evaluation, inspection of individual outcomes, and careful review and redaction of potentially sensitive exported session data.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.