A Beginner's Guide to Fine-tuning with Session Data + Fireworks

Captured coding-agent sessions can be converted into fine-tuning data by identifying a specific, measurable behavior that engineers repeatedly correct, separating agent-caused errors from changed requirements or newly supplied information, and labeling examples consistently with paperctl. Using 90 days of more than 500 sessions, the author found that context failures—where an agent failed to retrieve information available in files or documentation—were a high-cost target, while simply repeating context already visible to the agent did not improve outcomes. The proposed workflow is to collect both correction-and-recovery cases and positive examples of the desired first action, cut each session so only the target agent response is trained, export labeled traces into weighted JSONL training and evaluation files, sanitize them, and split held-out data by session. Before training, teams should define a decision matrix and success threshold covering first decisions, context checks, recovery, and tool execution, then compare a base model and a LoRA-fine-tuned Fireworks model under identical deployment settings. Results can be scored with blinded LLM judging, checked against manual reviews and executable tests, while real-world value should ultimately be measured through changes in rework spend after deployment; the post provides the pipeline for doing this rather than claiming that fine-tuning itself succeeds.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.