Introducing Supabase Evals

Overview of Supabase Evals

Supabase has introduced Supabase Evals, an open-source benchmarking framework aimed at evaluating the performance of AI coding agents when working with Supabase. This tool is designed to measure how effectively agents can complete real-world tasks, such as building a database schema, debugging a failed Edge Function, or fixing a broken RLS policy. The framework runs against a variety of AI agents, including Claude Code, Codex, and OpenCode, and scores their performance based on predefined criteria.

The results from this benchmark are used both in public reports and internal regression testing, providing a consistent way to monitor agent performance over time. As AI agents become a more common method for building with Supabase, the need for a reliable benchmarking system has become increasingly important.

Why Supabase Evals Was Built

Supabase Evals was developed to address the growing reliance on AI agents in the development process. With agents interacting through various Supabase tools, including the CLI, MCP server, agent skills, and documentation, there was a need to understand how well agents perform across these different interfaces. The framework allows for a comprehensive evaluation of agent behavior, ensuring that improvements and fixes are intentional and measurable.

By identifying where agents struggle, Supabase can make targeted improvements and verify that changes do not lead to regression. This approach also enables the testing of new features before they are released, ensuring a more stable and reliable development experience.

How the Framework Works

The framework is built around two main types of scenarios: benchmark and regression. Benchmark scenarios are designed to cover a broad range of real-world tasks, providing a diverse set of challenges that reflect the typical user journey. These scenarios are run against a variety of model configurations, and the results are published publicly.

Regression scenarios, on the other hand, focus on specific failure modes that are monitored more frequently. These scenarios are used to triage bug reports and experiment with new features without affecting the published benchmark results. Every scenario runs in a real Supabase environment, ensuring that the evaluation reflects actual agent behavior in production-like conditions.

The framework uses a combination of deterministic checks and LLM-as-a-judge evaluations to score agent performance. Deterministic checks assess whether specific outcomes are achieved, such as data access or function return values, while LLM-based evaluations are used for tasks requiring semantic judgment. Agents are allowed one retry after a failure to reduce false negatives and ensure the evaluation process remains sustainable.

Key Findings and Insights

Initial evaluations have revealed several important insights. One key finding is that the use of skills (predefined agent capabilities) has a significant impact on performance, particularly in edge cases where models need to unlearn outdated knowledge. Agents with skills loaded tend to consult Supabase documentation more consistently, leading to better performance in complex scenarios.

However, challenges remain. Agents often fail to use declarative schema workflows, which are designed to simplify database management. Additionally, newer Supabase libraries, such as @supabase/server, are not always discovered by agents, leading to suboptimal choices in certain tasks. These issues are being addressed through improved skill guidance and documentation clarification.

Another notable finding is that the use of documentation varies across different agent harnesses. Codex-based agents tend to consult documentation more frequently than Claude Code agents, and the most capable OpenAI models show the most consistent behavior. These differences highlight the need for further improvements in how agents interact with documentation and how they are trained to use it effectively.

Next Steps and Future Work

Supabase plans to expand the coverage of edge cases within the current benchmark and introduce new scenarios as both agents and the product evolve. The team is also working on refining the scoring system to make it more rigorous and stable. In the future, they aim to introduce a new CLI command and MCP tool that will allow agents to submit feedback when they encounter difficulties, helping to prioritize future improvements.

Supabase Evals is a starting point for evaluating AI agent performance in the Supabase ecosystem. As the framework continues to evolve, it will provide valuable insights into how agents can be improved and how Supabase can better support their use in development workflows.

Source: Supabase. Analysis based on publicly available information; no hands-on testing is claimed.