How Legora runs 2,000 parallel agent evals on Modal Sandboxes
Blog post from Modal
Legora rebuilt its evaluation system as it transitioned from a document-focused AI assistant to an autonomous legal agent capable of conducting hours-long work across thousands of documents. To achieve both scale and environment fidelity, it runs each evaluation in an isolated Modal Sandbox containing a commit-pinned version of its complete application stack, including document editing, databases, storage, PDF conversion, search infrastructure, and code-execution environments. Large experiments can involve 2,000 or more concurrent runs, with Modal Volumes providing shared document collections and reproducible snapshots of databases, permissions, and blobs, while image prewarming and caching reduce setup time. Legora adopted Modal VM Sandboxes after earlier container limitations required networking and filesystem workarounds, as full virtual machines allow Docker to operate like it does on standard Linux hosts. The resulting platform supports Legora’s internal benchmark for long-horizon legal tasks, guides model and agent-harness decisions, and enables engineers and agents to launch, analyze, and iteratively optimize experiments, including changes that reduced agent costs by 30 percent.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Secrets Management | 2 | 3,096 | 622 | 155 | +51% |
| AI Agents | 1 | 7,031 | 1,671 | 272 | +25% |
| Agent Evaluations | 1 | 168 | 81 | 34 | -20% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.