Are Supabase docs agent-friendly? We didn't know, so we checked

Testing for Agent-Friendliness in Supabase Documentation

Supabase has taken a proactive approach to evaluating its documentation by running an AI agent through a series of tasks to determine whether the guides are agent-friendly. This initiative was driven by the observation that users are increasingly relying on coding agents like Claude Code or Cursor to set up applications, rather than reading documentation manually. The challenge lies in ensuring that agents can accurately interpret and follow the instructions provided in the guides, as they may not detect subtle errors or omissions that a human reader might catch.

Methodology and Initial Findings

The evaluation focused on the Row Level Security (RLS) guide, a high-traffic page with a significant number of pageviews and a notable percentage of users reporting it as unhelpful. The team created an evaluation framework, or 'eval,' that runs an agent through a series of tasks and grades the results against a checklist. This checklist was not derived from the guide itself but from a broader understanding of what correct RLS implementation entails, including input from subject matter experts and external sources.

The initial runs of the eval revealed several issues, including the fact that the 'anon' role was granted write access to multiple tables, which is a security risk. This finding prompted the team to revise the documentation by adding procedural steps to revoke default grants and explicitly reapply policies where necessary. These changes had an immediate impact on the evaluation results, as the agent was now able to follow the correct steps without unintended access being granted.

Structural Adjustments for Better Agent Performance

While addressing the initial issues improved the results, the team discovered that agents often skipped sections of the documentation that were not directly relevant to the task at hand. This led to a structural adjustment, where test examples were moved closer to the code snippets that users would most likely copy and use. This change ensured that the guidance was more accessible to agents, as they were more likely to encounter the relevant information in the flow of their work.

The evaluation process also highlighted the importance of clear structure and explicit signposting in technical writing. Both agents and humans tend to read selectively, and the most effective approach for both audiences was to provide clear, well-organized content with obvious sections and instructions.

Broader Implications and Future Work

Supabase has now integrated the RLS evaluation into its public benchmark suite, and the team is expanding the use of evals to other popular guides. This ongoing process ensures that documentation remains aligned with the needs of both human and agent users. The initiative also underscores the importance of regular testing and iteration in documentation, as the needs of users and tools evolve over time.

The open-source nature of the evaluation framework, supabase/evals, allows for community contributions and further refinement of the approach. This initiative represents a significant step toward ensuring that Supabase documentation is not only accurate but also accessible and usable in an agent-driven development environment.

Source: Supabase. Analysis based on publicly available information; no hands-on testing is claimed.