> ## Content Index
> Fetch the complete content index at: https://www.stirup.si/llms.txt
> Use this file to discover other available public pages before exploring further.

# Parsimony: Benchmarking AI Code Generation by Code Quantity
- URL: https://www.stirup.si/parsimony-benchmarking-ai-code-generation-by-code-quantity-22dbce1d/
- Published: 2026-10-13T21:05:51.000Z
- Updated: 2026-10-13T21:05:51.000Z
- Description: Parsimony evaluates AI-generated code by measuring the amount of code added and removed, providing a nuanced view of code quality and maintainability.
- Author: Satish Das
- Tags: Community, News, Stirup Research

## Benchmarking AI Code Generation with Parsimony

Show HN: Parsimony – an open-source benchmark for how much code AI models add

Benchmarking AI Code Generation with Parsimony

Parsimony introduces a new approach to evaluating AI-generated code by focusing on the amount of code produced, rather than just the success rate or cost. Unlike existing benchmarks that measure solve rate and cost, Parsimony penalizes excess code, which can complicate a codebase over time. It analyzes published patches and counts coding units, such as statements, calls, comparisons, names, and literals, rather than lines of code. This method avoids rewarding overly verbose code or penalizing concise solutions. Additionally, Parsimony accounts for deleted code by subtracting it from the total, resulting in a net unit count that reflects the actual change made: net units = added − removed. This approach provides a more nuanced understanding of how AI models impact code quality and maintainability.

## Model Comparison and Leaderboard Mechanics

In the Model Comparison and Leaderboard Mechanics section of Parsimony, models are evaluated based on the mean net units of code they add when solving tasks. This metric is visualized through scatter plots and bar charts, with the X axis representing one metric and the Y axis another. Models are ranked by the lowest mean net units added, indicating efficiency in code modification. The leaderboard lists each task, model, and method, with patches applied to the task's code and both versions of every changed file parsed for analysis. Pass/fail outcomes are derived from SWE-bench's published results, and tests are not rerun, as noted in [Hacker News Show](https://parsimony.lasu.dev/?ref=stirup.si). This approach ensures transparency and consistency in benchmarking AI models, though it relies on existing test outcomes rather than new evaluations.

## Limitations and Correlations in Parsimony Metrics

Limitations and Correlations in Parsimony Metrics highlight key considerations when interpreting the benchmark's results. First, smaller code size does not necessarily equate to better readability or design, as the units used in Parsimony only measure size, not quality ([Hacker News Show](https://parsimony.lasu.dev/?ref=stirup.si)). Additionally, models that generate more code tend to solve more tasks, with a Spearman correlation ranging from 0.33 to 0.60 across boards with 26 or more models ([Hacker News Show](https://parsimony.lasu.dev/?ref=stirup.si)). This suggests a potential trade-off between code quantity and task-solving ability. Finally, while every model in the benchmark uses the same agent (mini-SWE-agent), the agent's influence on how much code is written remains a variable that can affect outcomes ([Hacker News Show](https://parsimony.lasu.dev/?ref=stirup.si)). These factors underscore the importance of interpreting Parsimony metrics with caution and considering them as one of many indicators of model performance.

## Sources and reporting notes

- [Hacker News Show](https://parsimony.lasu.dev/?ref=stirup.si)
- [agentmuster.app](https://agentmuster.app/?ref=stirup.si)

Based on published source material. No independent hands-on testing is claimed.