August 2026 Summaries

Supabase has open-sourced a tool called supabase/evals, designed to benchmark and assess the performance of AI agents, such as Claude Code and Codex, in handling real-world tasks within the Supabase ecosystem. This framework evaluates agents on tasks like building schemas and debugging Edge Functions, providing a scoring system that informs both a public benchmark and an internal regression suite. The initiative aims to measure the effectiveness of AI-driven development with Supabase, addressing the growing trend of using agents for project creation. By identifying areas where agents struggle, like inconsistent usage of declarative schemas and skill activation, Supabase seeks to refine its guidance and improve agent performance. The results are visualized on their website, offering insights into agent capabilities and areas needing improvement, with plans to enhance the tool further by expanding coverage and making scoring more stable.