HumanEval Benchmark Evaluates Code Generation Capabilities of AI Models
The HumanEval benchmark assesses models' ability to generate correct code from docstrings, with MiniCPM-SALA leading at 0.951, while GPT-3 scored 0%.
Overview of HumanEval Benchmark
The HumanEval benchmark assesses the functional correctness of models in synthesizing programs from docstrings, focusing on language comprehension, algorithms, and basic mathematics. Comprising 164 original programming problems, it serves as a text-based evaluation tool for reasoning and code generation tasks. According to the paper detailing the methodology and dataset construction, the benchmark provides a structured approach to measure model performance. In initial evaluations, a new model achieved a 28.8% success rate on the HumanEval set, significantly outperforming GPT-3, which solved 0% of the problems, and GPT-J, which solved 11.4%. These results highlight the benchmark's ability to differentiate between models based on their code synthesis capabilities.
Current Leaderboard and Model Performance
The current HumanEval leaderboard highlights MiniCPM-SALA from OpenBMB as the leading model, achieving a score of 0.951 according to llm-stats.com. This score places it at the top among 66 evaluated AI models, as noted in llm-stats.com. MiniCPM-SALA is also recognized as the top-ranked open-source model on HumanEval, with a score of 0.951 and a rank of #1, as stated in llm-stats.com. These results reflect the model's strong performance in code generation and reasoning tasks, though specific details about its training data or methodology are not provided in the available evidence.
Evaluation Details and Model Limitations
The HumanEval leaderboard on llm-stats.com highlights key evaluation details and model limitations. According to the evidence, careful investigation of models reveals challenges, such as difficulty in handling docstrings that describe long chains of operations and binding operations to variables . Additionally, the evaluation of 66 models on the HumanEval benchmark shows that 0 results have been verified, while 65 are self-reported . These findings underscore the importance of rigorous validation processes in assessing code generation capabilities. The leaderboard also touches on broader impacts of deploying such technologies, including considerations related to safety, security, and economics . These points emphasize the need for continued research and transparency in the development and deployment of powerful code generation models.
Cost and Performance Trade-offs
The HumanEval leaderboard, last updated in November 2026, includes 66 evaluated models, according to llm-stats.com. Among models scoring within 10% of the leader, Gemma 3 27B from Google stands out as the most cost-effective option, priced at $0.08 per million input tokens, with a score of 0.878. This highlights a key trade-off between cost and performance in large language models. While higher-performing models may offer better accuracy or efficiency, they often come with increased computational and financial costs. Users must weigh these factors based on their specific needs, whether for research, commercial applications, or general use. The leaderboard provides a valuable resource for comparing models, but it is essential to note that numeric claims are attributed to their respective publishers and should not be interpreted as universal guarantees.
Sources and reporting notes
Based on published source material. No independent hands-on testing is claimed.