July 2026 Summaries

In the described learning system, a frozen model acquires new capabilities through a cycle of detecting tasks it cannot perform, collecting attempts, and compiling successful results into cartridges using Prefix-Tuning methods, while the base model weights remain unchanged. The system begins with an empty library and grows to hold five new capabilities over six cycles, maintaining earlier learned functions even as new ones are added. Each cycle involves detecting missing knowledge, training, and validating the new capability before integrating it into the library, which is tested to ensure no damage to existing functions. The system's process of learning from its own traces involves verifying outputs against a deterministic checker and applying updates to real production traffic, achieving a win-or-tie rate increase from 42% to 59% over seven updates. Despite achieving these updates without performance degradation, the system faces challenges with selection accuracy, which significantly reduces the achievable success rate, highlighting the need for better selection mechanisms to enhance the system's efficiency and reliability.

The document explores the challenges and evaluation of a system designed to route queries to specialized models, known as cartridges, within a shared environment. The system aims to enhance the accuracy of selecting appropriate tools for given prompts but faces difficulties with routing precision as the number of tools increases. Initial tests show a significant drop in accuracy from 84.7% with two tools to 54.1% with forty, highlighting the complexity of managing large tool libraries. The system struggles with misrouting standard prompts to specialized tools, leading to a higher error rate than the predefined target of 2%. The architecture relies on vector-based routing, which proves ineffective as a primary dispatch method, necessitating a fallback to explicit capability tags and a version policy. The findings suggest that while improvements, such as using certainty thresholds, can enhance accuracy, the system still requires substantial standard query data and reveals significant risks in similar architectures. Despite efforts to refine the routing process, the system's performance often falls short of initial goals, indicating the need for more robust solutions in handling large-scale specialized model libraries.

The text explores various strategies for achieving successful task completion in reinforcement learning experiments, focusing on the limitations of using partial credit and memory systems in unsolved tasks. The experiments involve running an agent through tasks and using a testing suite to score performance, which highlights the difficulty of achieving a full success when relying solely on past failures or partial credit. Different approaches, such as guided exploration and high-temperature sampling, were tested but failed to consistently produce successful outcomes, emphasizing the need for a direct demonstration or imitation framework to improve agent performance. The document underscores the challenges of optimizing policies based on partial feedback and the necessity of starting with a successful example to guide learning. It also discusses how context bootstrapped reinforcement learning and supervised training on demonstrations can lead to better results. The experiments reveal that the model's architecture and task complexity significantly impact the ability to achieve complete task success, suggesting that providing a clear demonstration is a more effective strategy compared to relying on autonomous exploration in complex environments.

The text discusses an experimental approach to training models for task completion by examining the impact of different methodologies on performance, particularly focusing on provenance masking. This technique involves masking tokens based on their authorship, allowing models to learn from their errors without directly calculating loss from them. The study contrasts various training signals, including supervised fine-tuning (SFT) on successful sessions, observational training on agent transcripts, and interventional provenance-masked training. The research highlights the effectiveness of provenance masking in improving task-solving rates while reducing invalid actions, outperforming other methods such as full transcript training and critic-based selection. The text also delves into the challenges of incorporating new capabilities into models, emphasizing the need for separate training to achieve novel skills and the importance of carefully managing token authorship to optimize learning outcomes.

Ninety Days of Agent Work explores the intricacies of using different storage mechanisms for improving AI model performance, focusing on contexts, cartridges, and weights. The text highlights the transient nature of context as a storage space for instructions and the challenges in fine-tuning models, which affects all users. Cartridges, described as small add-ons, excel in encoding new capabilities without altering the base model, though they struggle with open-ended tasks. Weights, on the other hand, are better suited for complex and open-ended conversations due to their adaptability across requests. Experiments revealed that cartridges could achieve high success in novel tasks but face limitations in capacity and interference when combining multiple skills. The article also discusses the importance of maintaining task isolation and the potential for cartridges to degrade performance on unrelated tasks. It emphasizes the need for careful engineering to ensure optimal usage of these storage mechanisms, highlighting the trade-offs between cost, persistence, and cross-domain risks.

Between April 10 and July 10, 2026, engineers recorded 3,153 sessions with coding agents, generating 251,125 messages, primarily from the Claude Code tool, to study agent interaction and performance. This dataset, which largely consists of tool-generated text, was used to derive a 93% tool output metric, indicating that most of the content originates from the system rather than user prompts. During these sessions, engineers frequently intervened to correct the agents, with corrections occurring in 1,554 out of 2,007 labeled sessions, highlighting the importance of user feedback in improving agent skills. The study also identified a significant security finding, with 10% of sessions inadvertently leaking credentials, underscoring the necessity for careful data handling. The sessions revealed a 60% full resolution rate of tasks, while 28% were partially completed, providing valuable insights into agent performance and training potential. This comprehensive dataset, stored in a queryable format, offers significant potential for refining agent models and improving security practices within the engineering field.

The text discusses the challenges and methodologies involved in training machine learning models to retain new skills while preserving existing knowledge. It highlights the limitations of current practices where session data is discarded, leading to missed opportunities for improving models through the integration of post-deployment experiences. The document elaborates on various approaches, such as provenance masking and the use of cartridges, to enhance learning and skill retention. Provenance masking helps in selectively training models by excluding failed attempts, improving repair task success rates significantly. Cartridges are used to store discrete capabilities in models without interfering with existing skills, although they face limitations in retaining more than a few facts. The text also explores routing challenges within the system, where explicit tagging and routing mechanisms are necessary to optimize task-solving capabilities. The experiments highlight the importance of obtaining initial successes through external demonstrations to transition from failure to success, underscoring the need for a structured approach in updating machine learning models to balance the integration of new skills with the retention of existing ones.