Detailed results
100 repositories · 4.1k bugs · Updated November 2, 2026| Rank | Model | Agent | Bugs resolvedScore | Total USDUSD | USD / repo | Turns / repo | Tokens / repo | |
|---|---|---|---|---|---|---|---|---|
| 1 | Astra 6OpenAI | mini-SWE-agent | 9.5% | $3,255 | $35.00 | 113 | 119.5k | |
| 2 | Sol 5.6 (xhigh)OpenAI | mini-SWE-agent | 4.7% | $7,230 | $72.30 | 233 | 104.8k | |
| 3 | Opus 5.5 (xhigh)Anthropic | mini-SWE-agent | 3.6% | $2,198 | $21.98 | 266 | 112.7k | |
| 4 | Fable 5.1Anthropic | mini-SWE-agent | 2.9% | $3,904 | $39.04 | 319 | 165.0k | |
| 5 | Luna 5.6 (xhigh)OpenAI | mini-SWE-agent | 2.5% | $224 | $2.24 | 204 | 61.2k | |
| 6 | DeepSeek V4.1 Flash (max)DeepSeek | mini-SWE-agent | 1.7% | n/a | n/a | |||
| 7 | Terra 5.6 (xhigh)OpenAI | mini-SWE-agent | 1.5% | $357 | $3.57 | 76 | 46.6k | |
| 8 | Luna 5.6 (high)OpenAI | mini-SWE-agent | 1.4% | $28 | $0.28 | 75 | 20.3k | |
| 9 | Opus 5 (xhigh)Anthropic | mini-SWE-agent | 1.3% | $5,363 | $53.63 | 323 | 192.7k | |
| 10 | DeepSeek V4.1 Flash (high)DeepSeek | mini-SWE-agent | 0.9% | $95 | $0.97 | 347 | 241.1k | |
| 11 | GLM 5.3 Flash (max)Zhipu AI | mini-SWE-agent | 0.7% | n/a | n/a | |||
| 12 | MiniMax M3.1 Flash (max)MiniMax | mini-SWE-agent | 0.7% | $153 | $1.56 | 363 | 204.8k | |
| 13 | GLM 5.3 Flash (high)Zhipu AI | mini-SWE-agent | 0.6% | $177 | $1.81 | 332 | 115.6k | |
| 14 | ![]() |
Kimi K3Moonshot AI | mini-SWE-agent | 0.6% | $2,451 | $24.51 | 337 | 152.6k |
| 15 | Luna 5.6OpenAI | mini-SWE-agent | 0.5% | $4 | $0.04 | 22 | 4.5k | |
| 16 | GPT-5.4 Mini (high)OpenAI | mini-SWE-agent | 0.5% | $122 | $1.22 | 75 | 40.5k | |
| 17 | GPT-5.4 MiniOpenAI | mini-SWE-agent | 0.2% | $5 | $0.05 | 13 | 2.1k | |
| 18 | Gemini 3.5 Flash LiteGoogle | mini-SWE-agent | 0.1% | $6 | $0.06 | 34 | 5.2k |
Per-repository resource use includes non-deprecated retries and is averaged over evaluated repositories.
Pareto frontier
Hover a point for details · The line marks the Pareto frontier (best result per cost) · Click a point to see model details
Bugs resolved by model × repository
All models × 100 repositories
0%
100%
Hover for details · Click to open task
