Models
12
Mirantis TokenFactory · SWE-bench
Testing the latest open-weight models against SWE-bench
We ran 12 models through the same agent, the same gateway and the official SWE-bench grader. Every score links to the evidence behind it.
Three words we use
- Task
- One problem in the benchmark, a real bug report from an open-source project. The benchmark has 2,294 tasks.
- Attempt
- One model’s try at one task. 12 models x 2,294 tasks = 27,528 attempts.
- Solved
- The project’s own tests pass after the model’s patch.
Model names are the gateway's route names.
Tasks
2,294
Attempts
27,528
Attempts graded
27,528 (100%)
Charts need JavaScript. You can read every plotted value in the open tables below.
Resolved tasks
deepseek-v4.1-flash-1017 leads inside our limits, resolving 54.97% of the 2,294 Full tasks
The bars show what each model solved inside our limits, among its graded attempts. The lines show a 95% confidence interval (Wilson): a range describing uncertainty from the choice of tasks, not repeated attempts. GLM Flash routes use lighter fills. Diagonal marks identify uncensored or abliterated variants.
Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.
Show the data as a table
| Rank | Model | Resolved | Cut off by limits | Wilson interval | Resolved / graded | Coverage |
|---|---|---|---|---|---|---|
| 1 | deepseek-v4.1-flash-1017 | 54.97% | 5.01% (115 / 2,294) | 52.93%–57.00% | 1,261 / 2,294 | 100.00% |
| 2 | deepseek-v4.1-flash-1017-uncensored-fp8 | 52.62% | 5.19% (119 / 2,294) | 50.57%–54.65% | 1,207 / 2,294 | 100.00% |
| 3 | glm-5.3 | 52.01% | 5.32% (122 / 2,294) | 49.96%–54.04% | 1,193 / 2,294 | 100.00% |
| 4 | glm-5.3-flash-uncensored-fp8 | 50.09% | 8.15% (187 / 2,294) | 48.04%–52.13% | 1,149 / 2,294 | 100.00% |
| 5 | kimi-k3 | 49.30% | 22.80% (523 / 2,294) · limit-constrained | 47.26%–51.35% | 1,131 / 2,294 | 100.00% |
| 6 | glm-5.3-flash | 48.69% | 11.60% (266 / 2,294) | 46.65%–50.74% | 1,117 / 2,294 | 100.00% |
| 7 | nemotron-3-ultra | 40.76% | 4.36% (100 / 2,294) | 38.76%–42.78% | 935 / 2,294 | 100.00% |
| 8 | kimi-k3-abliterated | 38.32% | 45.03% (1,033 / 2,294) · limit-constrained | 36.35%–40.32% | 879 / 2,294 | 100.00% |
| 9 | glm-5.3-uncensored-fp8 | 34.09% | 45.95% (1,054 / 2,294) · limit-constrained | 32.18%–36.05% | 782 / 2,294 | 100.00% |
| 10 | nemotron-3-super | 31.52% | 6.36% (146 / 2,294) | 29.65%–33.45% | 723 / 2,294 | 100.00% |
| 11 | nemotron-3.5-lightning | 17.00% | 6.41% (147 / 2,294) | 15.52%–18.59% | 390 / 2,294 | 100.00% |
| 12 | minimax-m3.1-flash | 16.39% | 75.50% (1,732 / 2,294) · limit-constrained | 14.93%–17.96% | 376 / 2,294 | 100.00% |
Source: summary.json and subsets.json.
Show the data as a table
| Rank | Model | Resolved | Cut off by limits | Wilson interval | Resolved / graded | Coverage |
|---|---|---|---|---|---|---|
| 1 | deepseek-v4.1-flash-1017 | 79.40% | 2.40% (12 / 500) | 75.64%–82.71% | 397 / 500 | 100.00% |
| 2 | glm-5.3 | 78.80% | 1.80% (9 / 500) | 75.00%–82.16% | 394 / 500 | 100.00% |
| 3 | deepseek-v4.1-flash-1017-uncensored-fp8 | 77.00% | 2.60% (13 / 500) | 73.11%–80.47% | 385 / 500 | 100.00% |
| 4 | glm-5.3-flash-uncensored-fp8 | 76.20% | 3.40% (17 / 500) | 72.28%–79.72% | 381 / 500 | 100.00% |
| 5 | glm-5.3-flash | 74.00% | 7.20% (36 / 500) | 69.98%–77.65% | 370 / 500 | 100.00% |
| 6 | kimi-k3 | 72.60% | 14.20% (71 / 500) | 68.53%–76.33% | 363 / 500 | 100.00% |
| 7 | nemotron-3-ultra | 68.00% | 2.20% (11 / 500) | 63.79%–71.94% | 340 / 500 | 100.00% |
| 8 | kimi-k3-abliterated | 62.20% | 30.80% (154 / 500) · limit-constrained | 57.87%–66.34% | 311 / 500 | 100.00% |
| 9 | nemotron-3-super | 56.80% | 4.80% (24 / 500) | 52.42%–61.07% | 284 / 500 | 100.00% |
| 10 | glm-5.3-uncensored-fp8 | 56.20% | 33.20% (166 / 500) · limit-constrained | 51.82%–60.49% | 281 / 500 | 100.00% |
| 11 | nemotron-3.5-lightning | 29.20% | 8.20% (41 / 500) | 25.39%–33.33% | 146 / 500 | 100.00% |
| 12 | minimax-m3.1-flash | 28.60% | 66.20% (331 / 500) · limit-constrained | 24.81%–32.71% | 143 / 500 | 100.00% |
Source: summary.json and subsets.json.
Show the data as a table
| Rank | Model | Resolved | Cut off by limits | Wilson interval | Resolved / graded | Coverage |
|---|---|---|---|---|---|---|
| 1 | deepseek-v4.1-flash-1017 | 66.33% | 2.00% (6 / 300) | 60.81%–71.44% | 199 / 300 | 100.00% |
| 1 | glm-5.3 | 66.33% | 2.00% (6 / 300) | 60.81%–71.44% | 199 / 300 | 100.00% |
| 3 | deepseek-v4.1-flash-1017-uncensored-fp8 | 65.67% | 1.67% (5 / 300) | 60.13%–70.81% | 197 / 300 | 100.00% |
| 4 | glm-5.3-flash | 61.67% | 7.33% (22 / 300) | 56.05%–66.99% | 185 / 300 | 100.00% |
| 4 | glm-5.3-flash-uncensored-fp8 | 61.67% | 5.67% (17 / 300) | 56.05%–66.99% | 185 / 300 | 100.00% |
| 6 | kimi-k3 | 60.67% | 17.00% (51 / 300) | 55.04%–66.03% | 182 / 300 | 100.00% |
| 7 | nemotron-3-ultra | 53.67% | 1.67% (5 / 300) | 48.01%–59.23% | 161 / 300 | 100.00% |
| 8 | kimi-k3-abliterated | 49.67% | 36.67% (110 / 300) · limit-constrained | 44.05%–55.29% | 149 / 300 | 100.00% |
| 9 | glm-5.3-uncensored-fp8 | 47.33% | 36.00% (108 / 300) · limit-constrained | 41.75%–52.98% | 142 / 300 | 100.00% |
| 10 | nemotron-3-super | 44.00% | 6.00% (18 / 300) | 38.49%–49.66% | 132 / 300 | 100.00% |
| 11 | minimax-m3.1-flash | 22.33% | 69.33% (208 / 300) · limit-constrained | 17.99%–27.38% | 67 / 300 | 100.00% |
| 11 | nemotron-3.5-lightning | 22.33% | 5.33% (16 / 300) | 17.99%–27.38% | 67 / 300 | 100.00% |
Source: summary.json and subsets.json.
Based on 12 models x 2,294 tasks = 27,528 attempts.
Attempt outcomes
5,594 attempts ended with no patch, and 120 ran out of time
All 2,294 tasks for every model. Colour shows how each attempt ended, not which model made it. Each attempt is counted once, in the first group that fits: solved, still waiting to be graded, ran out of time, no patch submitted, or a patch that failed the tests. See the methodology section on limits and their effects.
Limits and their effects · How we count timeouts
Show the data as a table
| Model | Solved | Patch failed the tests | No patch submitted | Ran out of time | Total |
|---|---|---|---|---|---|
| deepseek-v4.1-flash-1017 | 1,261 (54.97%) | 913 (39.80%) | 120 (5.23%) | 0 (0.00%) | 2,294 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | 1,207 (52.62%) | 968 (42.20%) | 113 (4.93%) | 6 (0.26%) | 2,294 |
| glm-5.3 | 1,193 (52.01%) | 977 (42.59%) | 124 (5.41%) | 0 (0.00%) | 2,294 |
| glm-5.3-flash-uncensored-fp8 | 1,149 (50.09%) | 957 (41.72%) | 188 (8.20%) | 0 (0.00%) | 2,294 |
| kimi-k3 | 1,131 (49.30%) | 638 (27.81%) | 497 (21.67%) | 28 (1.22%) | 2,294 |
| glm-5.3-flash | 1,117 (48.69%) | 911 (39.71%) | 253 (11.03%) | 13 (0.57%) | 2,294 |
| nemotron-3-ultra | 935 (40.76%) | 1,225 (53.40%) | 133 (5.80%) | 1 (0.04%) | 2,294 |
| kimi-k3-abliterated | 879 (38.32%) | 382 (16.65%) | 962 (41.94%) | 71 (3.10%) | 2,294 |
| glm-5.3-uncensored-fp8 | 782 (34.09%) | 442 (19.27%) | 1,070 (46.64%) | 0 (0.00%) | 2,294 |
| nemotron-3-super | 723 (31.52%) | 1,363 (59.42%) | 207 (9.02%) | 1 (0.04%) | 2,294 |
| nemotron-3.5-lightning | 390 (17.00%) | 1,711 (74.59%) | 193 (8.41%) | 0 (0.00%) | 2,294 |
| minimax-m3.1-flash | 376 (16.39%) | 184 (8.02%) | 1,734 (75.59%) | 0 (0.00%) | 2,294 |
Source: instances.csv, grade reasons and timeout-adjudications.json.
Based on 12 models x 2,294 tasks = 27,528 attempts.
How many tasks the models can solve together
All 12 models together solve 1,586 of the 2,294 tasks (69.1%)
Here is what happens when you combine the models. The best single model solves 1,261 tasks. Adding the other 11 brings in only 325 more. The remaining 708 tasks, 30.9% of the benchmark, were solved by no model.
Each step adds one model. Scroll across the chart on a small screen to read every model.
Show the data as a table
| Number of models combined | Model added | New tasks solved | Tasks solved by at least one model |
|---|---|---|---|
| 0 | No models | 0 | 0 |
| 1 | deepseek-v4.1-flash-1017 | 1,261 | 1,261 |
| 2 | deepseek-v4.1-flash-1017-uncensored-fp8 | 93 | 1,354 |
| 3 | glm-5.3 | 77 | 1,431 |
| 4 | glm-5.3-flash-uncensored-fp8 | 35 | 1,466 |
| 5 | kimi-k3 | 66 | 1,532 |
| 6 | glm-5.3-flash | 16 | 1,548 |
| 7 | nemotron-3-ultra | 6 | 1,554 |
| 8 | kimi-k3-abliterated | 20 | 1,574 |
| 9 | glm-5.3-uncensored-fp8 | 4 | 1,578 |
| 10 | nemotron-3-super | 2 | 1,580 |
| 11 | nemotron-3.5-lightning | 4 | 1,584 |
| 12 | minimax-m3.1-flash | 2 | 1,586 |
Source: instances.csv and public SWE-bench issue text.
Show the data as a table
| Number of models combined | Model added | New tasks solved | Tasks solved by at least one model |
|---|---|---|---|
| 0 | No models | 0 | 0 |
| 1 | deepseek-v4.1-flash-1017 | 1,261 | 1,261 |
| 2 | kimi-k3 | 154 | 1,415 |
| 3 | glm-5.3 | 65 | 1,480 |
| 4 | glm-5.3-flash-uncensored-fp8 | 35 | 1,515 |
| 5 | kimi-k3-abliterated | 24 | 1,539 |
| 6 | glm-5.3-flash | 17 | 1,556 |
| 7 | deepseek-v4.1-flash-1017-uncensored-fp8 | 13 | 1,569 |
| 8 | nemotron-3-ultra | 5 | 1,574 |
| 9 | nemotron-3.5-lightning | 5 | 1,579 |
| 10 | glm-5.3-uncensored-fp8 | 4 | 1,583 |
| 11 | minimax-m3.1-flash | 2 | 1,585 |
| 12 | nemotron-3-super | 1 | 1,586 |
Source: instances.csv and public SWE-bench issue text.
astropy__astropy-11693
astropy/astropy
'WCS.all_world2pix' failed to converge when plotting WCS with non linear distortions
astropy__astropy-12907
astropy/astropy
Modeling's `separability_matrix` does not compute separability correctly for nested CompoundModels
Based on 12 models x 2,294 tasks = 27,528 attempts.
Where models differ · Full
Some models solve tasks no other model can
These are tasks where exactly one of the 12 models passed the tests. They show where models differ, not just who scores higher.
Show the data as a table
| Model | Tasks no other model solved |
|---|---|
| deepseek-v4.1-flash-1017 | 22 tasks |
| deepseek-v4.1-flash-1017-uncensored-fp8 | 11 tasks |
| glm-5.3 | 23 tasks |
| glm-5.3-flash-uncensored-fp8 | 12 tasks |
| kimi-k3 | 34 tasks |
| glm-5.3-flash | 14 tasks |
| nemotron-3-ultra | 3 tasks |
| kimi-k3-abliterated | 19 tasks |
| glm-5.3-uncensored-fp8 | 4 tasks |
| nemotron-3-super | 1 tasks |
| nemotron-3.5-lightning | 4 tasks |
| minimax-m3.1-flash | 2 tasks |
Source: instances.csv.
Based on 12 models x 2,294 tasks = 27,528 attempts.
Where models win and lose
The hardest project was requests, a library for making web requests: the typical model solved 25.00% of its 44 tasks
Each row is a model and each column is one of the 12 open-source projects the tasks come from. A cell shows the share of that project's tasks the model solved, and darker means more. Hover a cell for the exact counts.
Show the data as a table
| Model | Repository | Resolved | Resolved / total | Graded / total |
|---|---|---|---|---|
| deepseek-v4.1-flash-1017 | astropy/astropy | 47.37% | 45 / 95 | 95 / 95 |
| deepseek-v4.1-flash-1017 | django/django | 62.82% | 534 / 850 | 850 / 850 |
| deepseek-v4.1-flash-1017 | matplotlib/matplotlib | 68.48% | 126 / 184 | 184 / 184 |
| deepseek-v4.1-flash-1017 | mwaskom/seaborn | 59.09% | 13 / 22 | 22 / 22 |
| deepseek-v4.1-flash-1017 | pallets/flask | 72.73% | 8 / 11 | 11 / 11 |
| deepseek-v4.1-flash-1017 | psf/requests | 27.27% | 12 / 44 | 44 / 44 |
| deepseek-v4.1-flash-1017 | pydata/xarray | 50.91% | 56 / 110 | 110 / 110 |
| deepseek-v4.1-flash-1017 | pylint-dev/pylint | 35.09% | 20 / 57 | 57 / 57 |
| deepseek-v4.1-flash-1017 | pytest-dev/pytest | 55.46% | 66 / 119 | 119 / 119 |
| deepseek-v4.1-flash-1017 | scikit-learn/scikit-learn | 50.22% | 115 / 229 | 229 / 229 |
| deepseek-v4.1-flash-1017 | sphinx-doc/sphinx | 50.80% | 95 / 187 | 187 / 187 |
| deepseek-v4.1-flash-1017 | sympy/sympy | 44.30% | 171 / 386 | 386 / 386 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | astropy/astropy | 44.21% | 42 / 95 | 95 / 95 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | django/django | 61.29% | 521 / 850 | 850 / 850 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | matplotlib/matplotlib | 61.41% | 113 / 184 | 184 / 184 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | mwaskom/seaborn | 59.09% | 13 / 22 | 22 / 22 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | pallets/flask | 72.73% | 8 / 11 | 11 / 11 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | psf/requests | 29.55% | 13 / 44 | 44 / 44 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | pydata/xarray | 44.55% | 49 / 110 | 110 / 110 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | pylint-dev/pylint | 36.84% | 21 / 57 | 57 / 57 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | pytest-dev/pytest | 55.46% | 66 / 119 | 119 / 119 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | scikit-learn/scikit-learn | 48.91% | 112 / 229 | 229 / 229 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | sphinx-doc/sphinx | 49.20% | 92 / 187 | 187 / 187 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | sympy/sympy | 40.67% | 157 / 386 | 386 / 386 |
| glm-5.3 | astropy/astropy | 48.42% | 46 / 95 | 95 / 95 |
| glm-5.3 | django/django | 57.41% | 488 / 850 | 850 / 850 |
| glm-5.3 | matplotlib/matplotlib | 60.87% | 112 / 184 | 184 / 184 |
| glm-5.3 | mwaskom/seaborn | 50.00% | 11 / 22 | 22 / 22 |
| glm-5.3 | pallets/flask | 54.55% | 6 / 11 | 11 / 11 |
| glm-5.3 | psf/requests | 27.27% | 12 / 44 | 44 / 44 |
| glm-5.3 | pydata/xarray | 42.73% | 47 / 110 | 110 / 110 |
| glm-5.3 | pylint-dev/pylint | 38.60% | 22 / 57 | 57 / 57 |
| glm-5.3 | pytest-dev/pytest | 55.46% | 66 / 119 | 119 / 119 |
| glm-5.3 | scikit-learn/scikit-learn | 50.22% | 115 / 229 | 229 / 229 |
| glm-5.3 | sphinx-doc/sphinx | 50.27% | 94 / 187 | 187 / 187 |
| glm-5.3 | sympy/sympy | 45.08% | 174 / 386 | 386 / 386 |
| glm-5.3-flash-uncensored-fp8 | astropy/astropy | 42.11% | 40 / 95 | 95 / 95 |
| glm-5.3-flash-uncensored-fp8 | django/django | 56.71% | 482 / 850 | 850 / 850 |
| glm-5.3-flash-uncensored-fp8 | matplotlib/matplotlib | 54.89% | 101 / 184 | 184 / 184 |
| glm-5.3-flash-uncensored-fp8 | mwaskom/seaborn | 45.45% | 10 / 22 | 22 / 22 |
| glm-5.3-flash-uncensored-fp8 | pallets/flask | 54.55% | 6 / 11 | 11 / 11 |
| glm-5.3-flash-uncensored-fp8 | psf/requests | 34.09% | 15 / 44 | 44 / 44 |
| glm-5.3-flash-uncensored-fp8 | pydata/xarray | 41.82% | 46 / 110 | 110 / 110 |
| glm-5.3-flash-uncensored-fp8 | pylint-dev/pylint | 36.84% | 21 / 57 | 57 / 57 |
| glm-5.3-flash-uncensored-fp8 | pytest-dev/pytest | 54.62% | 65 / 119 | 119 / 119 |
| glm-5.3-flash-uncensored-fp8 | scikit-learn/scikit-learn | 48.03% | 110 / 229 | 229 / 229 |
| glm-5.3-flash-uncensored-fp8 | sphinx-doc/sphinx | 49.73% | 93 / 187 | 187 / 187 |
| glm-5.3-flash-uncensored-fp8 | sympy/sympy | 41.45% | 160 / 386 | 386 / 386 |
| kimi-k3 | astropy/astropy | 25.26% | 24 / 95 | 95 / 95 |
| kimi-k3 | django/django | 59.53% | 506 / 850 | 850 / 850 |
| kimi-k3 | matplotlib/matplotlib | 52.72% | 97 / 184 | 184 / 184 |
| kimi-k3 | mwaskom/seaborn | 45.45% | 10 / 22 | 22 / 22 |
| kimi-k3 | pallets/flask | 72.73% | 8 / 11 | 11 / 11 |
| kimi-k3 | psf/requests | 27.27% | 12 / 44 | 44 / 44 |
| kimi-k3 | pydata/xarray | 41.82% | 46 / 110 | 110 / 110 |
| kimi-k3 | pylint-dev/pylint | 33.33% | 19 / 57 | 57 / 57 |
| kimi-k3 | pytest-dev/pytest | 53.78% | 64 / 119 | 119 / 119 |
| kimi-k3 | scikit-learn/scikit-learn | 54.59% | 125 / 229 | 229 / 229 |
| kimi-k3 | sphinx-doc/sphinx | 45.45% | 85 / 187 | 187 / 187 |
| kimi-k3 | sympy/sympy | 34.97% | 135 / 386 | 386 / 386 |
| glm-5.3-flash | astropy/astropy | 45.26% | 43 / 95 | 95 / 95 |
| glm-5.3-flash | django/django | 53.65% | 456 / 850 | 850 / 850 |
| glm-5.3-flash | matplotlib/matplotlib | 55.98% | 103 / 184 | 184 / 184 |
| glm-5.3-flash | mwaskom/seaborn | 45.45% | 10 / 22 | 22 / 22 |
| glm-5.3-flash | pallets/flask | 54.55% | 6 / 11 | 11 / 11 |
| glm-5.3-flash | psf/requests | 27.27% | 12 / 44 | 44 / 44 |
| glm-5.3-flash | pydata/xarray | 39.09% | 43 / 110 | 110 / 110 |
| glm-5.3-flash | pylint-dev/pylint | 36.84% | 21 / 57 | 57 / 57 |
| glm-5.3-flash | pytest-dev/pytest | 55.46% | 66 / 119 | 119 / 119 |
| glm-5.3-flash | scikit-learn/scikit-learn | 44.54% | 102 / 229 | 229 / 229 |
| glm-5.3-flash | sphinx-doc/sphinx | 47.59% | 89 / 187 | 187 / 187 |
| glm-5.3-flash | sympy/sympy | 43.01% | 166 / 386 | 386 / 386 |
| nemotron-3-ultra | astropy/astropy | 37.89% | 36 / 95 | 95 / 95 |
| nemotron-3-ultra | django/django | 47.06% | 400 / 850 | 850 / 850 |
| nemotron-3-ultra | matplotlib/matplotlib | 44.02% | 81 / 184 | 184 / 184 |
| nemotron-3-ultra | mwaskom/seaborn | 40.91% | 9 / 22 | 22 / 22 |
| nemotron-3-ultra | pallets/flask | 45.45% | 5 / 11 | 11 / 11 |
| nemotron-3-ultra | psf/requests | 18.18% | 8 / 44 | 44 / 44 |
| nemotron-3-ultra | pydata/xarray | 36.36% | 40 / 110 | 110 / 110 |
| nemotron-3-ultra | pylint-dev/pylint | 31.58% | 18 / 57 | 57 / 57 |
| nemotron-3-ultra | pytest-dev/pytest | 45.38% | 54 / 119 | 119 / 119 |
| nemotron-3-ultra | scikit-learn/scikit-learn | 39.30% | 90 / 229 | 229 / 229 |
| nemotron-3-ultra | sphinx-doc/sphinx | 31.02% | 58 / 187 | 187 / 187 |
| nemotron-3-ultra | sympy/sympy | 35.23% | 136 / 386 | 386 / 386 |
| kimi-k3-abliterated | astropy/astropy | 38.95% | 37 / 95 | 95 / 95 |
| kimi-k3-abliterated | django/django | 51.65% | 439 / 850 | 850 / 850 |
| kimi-k3-abliterated | matplotlib/matplotlib | 30.43% | 56 / 184 | 184 / 184 |
| kimi-k3-abliterated | mwaskom/seaborn | 9.09% | 2 / 22 | 22 / 22 |
| kimi-k3-abliterated | pallets/flask | 45.45% | 5 / 11 | 11 / 11 |
| kimi-k3-abliterated | psf/requests | 22.73% | 10 / 44 | 44 / 44 |
| kimi-k3-abliterated | pydata/xarray | 26.36% | 29 / 110 | 110 / 110 |
| kimi-k3-abliterated | pylint-dev/pylint | 12.28% | 7 / 57 | 57 / 57 |
| kimi-k3-abliterated | pytest-dev/pytest | 36.13% | 43 / 119 | 119 / 119 |
| kimi-k3-abliterated | scikit-learn/scikit-learn | 35.81% | 82 / 229 | 229 / 229 |
| kimi-k3-abliterated | sphinx-doc/sphinx | 33.16% | 62 / 187 | 187 / 187 |
| kimi-k3-abliterated | sympy/sympy | 27.72% | 107 / 386 | 386 / 386 |
| glm-5.3-uncensored-fp8 | astropy/astropy | 28.42% | 27 / 95 | 95 / 95 |
| glm-5.3-uncensored-fp8 | django/django | 41.65% | 354 / 850 | 850 / 850 |
| glm-5.3-uncensored-fp8 | matplotlib/matplotlib | 38.59% | 71 / 184 | 184 / 184 |
| glm-5.3-uncensored-fp8 | mwaskom/seaborn | 36.36% | 8 / 22 | 22 / 22 |
| glm-5.3-uncensored-fp8 | pallets/flask | 54.55% | 6 / 11 | 11 / 11 |
| glm-5.3-uncensored-fp8 | psf/requests | 20.45% | 9 / 44 | 44 / 44 |
| glm-5.3-uncensored-fp8 | pydata/xarray | 33.64% | 37 / 110 | 110 / 110 |
| glm-5.3-uncensored-fp8 | pylint-dev/pylint | 15.79% | 9 / 57 | 57 / 57 |
| glm-5.3-uncensored-fp8 | pytest-dev/pytest | 38.66% | 46 / 119 | 119 / 119 |
| glm-5.3-uncensored-fp8 | scikit-learn/scikit-learn | 34.50% | 79 / 229 | 229 / 229 |
| glm-5.3-uncensored-fp8 | sphinx-doc/sphinx | 25.13% | 47 / 187 | 187 / 187 |
| glm-5.3-uncensored-fp8 | sympy/sympy | 23.06% | 89 / 386 | 386 / 386 |
| nemotron-3-super | astropy/astropy | 29.47% | 28 / 95 | 95 / 95 |
| nemotron-3-super | django/django | 39.18% | 333 / 850 | 850 / 850 |
| nemotron-3-super | matplotlib/matplotlib | 28.26% | 52 / 184 | 184 / 184 |
| nemotron-3-super | mwaskom/seaborn | 36.36% | 8 / 22 | 22 / 22 |
| nemotron-3-super | pallets/flask | 18.18% | 2 / 11 | 11 / 11 |
| nemotron-3-super | psf/requests | 13.64% | 6 / 44 | 44 / 44 |
| nemotron-3-super | pydata/xarray | 29.09% | 32 / 110 | 110 / 110 |
| nemotron-3-super | pylint-dev/pylint | 22.81% | 13 / 57 | 57 / 57 |
| nemotron-3-super | pytest-dev/pytest | 33.61% | 40 / 119 | 119 / 119 |
| nemotron-3-super | scikit-learn/scikit-learn | 27.95% | 64 / 229 | 229 / 229 |
| nemotron-3-super | sphinx-doc/sphinx | 23.53% | 44 / 187 | 187 / 187 |
| nemotron-3-super | sympy/sympy | 26.17% | 101 / 386 | 386 / 386 |
| nemotron-3.5-lightning | astropy/astropy | 15.79% | 15 / 95 | 95 / 95 |
| nemotron-3.5-lightning | django/django | 15.18% | 129 / 850 | 850 / 850 |
| nemotron-3.5-lightning | matplotlib/matplotlib | 23.37% | 43 / 184 | 184 / 184 |
| nemotron-3.5-lightning | mwaskom/seaborn | 22.73% | 5 / 22 | 22 / 22 |
| nemotron-3.5-lightning | pallets/flask | 27.27% | 3 / 11 | 11 / 11 |
| nemotron-3.5-lightning | psf/requests | 11.36% | 5 / 44 | 44 / 44 |
| nemotron-3.5-lightning | pydata/xarray | 17.27% | 19 / 110 | 110 / 110 |
| nemotron-3.5-lightning | pylint-dev/pylint | 7.02% | 4 / 57 | 57 / 57 |
| nemotron-3.5-lightning | pytest-dev/pytest | 23.53% | 28 / 119 | 119 / 119 |
| nemotron-3.5-lightning | scikit-learn/scikit-learn | 18.34% | 42 / 229 | 229 / 229 |
| nemotron-3.5-lightning | sphinx-doc/sphinx | 13.37% | 25 / 187 | 187 / 187 |
| nemotron-3.5-lightning | sympy/sympy | 18.65% | 72 / 386 | 386 / 386 |
| minimax-m3.1-flash | astropy/astropy | 22.11% | 21 / 95 | 95 / 95 |
| minimax-m3.1-flash | django/django | 20.94% | 178 / 850 | 850 / 850 |
| minimax-m3.1-flash | matplotlib/matplotlib | 18.48% | 34 / 184 | 184 / 184 |
| minimax-m3.1-flash | mwaskom/seaborn | 18.18% | 4 / 22 | 22 / 22 |
| minimax-m3.1-flash | pallets/flask | 36.36% | 4 / 11 | 11 / 11 |
| minimax-m3.1-flash | psf/requests | 15.91% | 7 / 44 | 44 / 44 |
| minimax-m3.1-flash | pydata/xarray | 18.18% | 20 / 110 | 110 / 110 |
| minimax-m3.1-flash | pylint-dev/pylint | 10.53% | 6 / 57 | 57 / 57 |
| minimax-m3.1-flash | pytest-dev/pytest | 12.61% | 15 / 119 | 119 / 119 |
| minimax-m3.1-flash | scikit-learn/scikit-learn | 13.54% | 31 / 229 | 229 / 229 |
| minimax-m3.1-flash | sphinx-doc/sphinx | 5.88% | 11 / 187 | 187 / 187 |
| minimax-m3.1-flash | sympy/sympy | 11.66% | 45 / 386 | 386 / 386 |
Source: repositories.json and instances.csv.
What the 12 projects are
| Project | What it is | Number of tasks |
|---|---|---|
| astropy/astropy | astronomy tools for Python | 95 |
| django/django | a web framework | 850 |
| matplotlib/matplotlib | a charting and plotting library | 184 |
| mwaskom/seaborn | statistical charts built on matplotlib | 22 |
| pallets/flask | a small web framework | 11 |
| psf/requests | a library for making web requests | 44 |
| pydata/xarray | labelled multi-dimensional data arrays | 110 |
| pylint-dev/pylint | a code checker | 57 |
| pytest-dev/pytest | a testing tool | 119 |
| scikit-learn/scikit-learn | a machine learning library | 229 |
| sphinx-doc/sphinx | a documentation builder | 187 |
| sympy/sympy | a library for symbolic maths | 386 |
Based on 12 models x 2,294 tasks = 27,528 attempts.
Steps versus score
More steps did not mean better scores
Each dot is one model. Further left means it usually needed fewer steps per attempt; higher up means it solved more of the 2,294 tasks. A step is one round trip: the model asks for a command, the tool runs it, and the output goes back.
The two highest-scoring models took about 22 and 23 steps per attempt; the model that took the most steps (74) solved 17.00% of the 2,294 tasks. minimax-m3.1-flash took few steps but solved few tasks; most of its attempts ended with no patch (75.59% of its 2,294 attempts).
Show the data as a table
| Model | Steps per attempt (median) | Solved | Attempts with steps |
|---|---|---|---|
| deepseek-v4.1-flash-1017 | 22 | 54.97% | 2,294 |
| deepseek-v4.1-flash-1017-uncensored-fp8 | 23 | 52.62% | 2,294 |
| glm-5.3 | 34.5 | 52.01% | 2,294 |
| glm-5.3-flash-uncensored-fp8 | 28 | 50.09% | 2,294 |
| kimi-k3 | 34 | 49.30% | 2,294 |
| glm-5.3-flash | 27.5 | 48.69% | 2,294 |
| nemotron-3-ultra | 63 | 40.76% | 2,294 |
| kimi-k3-abliterated | 28 | 38.32% | 2,294 |
| glm-5.3-uncensored-fp8 | 31 | 34.09% | 2,294 |
| nemotron-3-super | 54 | 31.52% | 2,294 |
| nemotron-3.5-lightning | 74 | 17.00% | 2,294 |
| minimax-m3.1-flash | 30 | 16.39% | 2,294 |
Source: trajectory model_stats.api_calls and summary.json.
Based on 12 models x 2,294 tasks = 27,528 attempts. Step counts are available for 27,528 inspected attempts.
Leaderboard data
| 1 | deepseek-v4.1-flash-1017 | 54.97%52.93%–57.00% | 5.01% (115 / 2,294) | 1,261 / 2,294 | 100.00% | 94.77%2,294 outcomes observed | 222,294 trajectories | Final |
|---|---|---|---|---|---|---|---|---|
| 2 | deepseek-v4.1-flash-1017-uncensored-fp8 | 52.62%50.57%–54.65% | 5.19% (119 / 2,294) | 1,207 / 2,294 | 100.00% | 94.81%2,294 outcomes observed | 232,294 trajectories | Final |
| 3 | glm-5.3 | 52.01%49.96%–54.04% | 5.32% (122 / 2,294) | 1,193 / 2,294 | 100.00% | 94.64%2,294 outcomes observed | 34.52,294 trajectories | Final |
| 4 | glm-5.3-flash-uncensored-fp8 | 50.09%48.04%–52.13% | 8.15% (187 / 2,294) | 1,149 / 2,294 | 100.00% | 91.80%2,294 outcomes observed | 282,294 trajectories | Final |
| 5 | kimi-k3 | 49.30%47.26%–51.35% | 22.80% (523 / 2,294) · limit-constrained | 1,131 / 2,294 | 100.00% | 77.11%2,294 outcomes observed | 342,294 trajectories | Final |
| 6 | glm-5.3-flash | 48.69%46.65%–50.74% | 11.60% (266 / 2,294) | 1,117 / 2,294 | 100.00% | 88.40%2,294 outcomes observed | 27.52,294 trajectories | Final |
| 7 | nemotron-3-ultra | 40.76%38.76%–42.78% | 4.36% (100 / 2,294) | 935 / 2,294 | 100.00% | 94.20%2,294 outcomes observed | 632,294 trajectories | Final |
| 8 | kimi-k3-abliterated | 38.32%36.35%–40.32% | 45.03% (1,033 / 2,294) · limit-constrained | 879 / 2,294 | 100.00% | 54.97%2,294 outcomes observed | 282,294 trajectories | Final |
| 9 | glm-5.3-uncensored-fp8 | 34.09%32.18%–36.05% | 45.95% (1,054 / 2,294) · limit-constrained | 782 / 2,294 | 100.00% | 53.36%2,294 outcomes observed | 312,294 trajectories | Final |
| 10 | nemotron-3-super | 31.52%29.65%–33.45% | 6.36% (146 / 2,294) | 723 / 2,294 | 100.00% | 93.11%2,294 outcomes observed | 542,294 trajectories | Final |
| 11 | nemotron-3.5-lightning | 17.00%15.52%–18.59% | 6.41% (147 / 2,294) | 390 / 2,294 | 100.00% | 93.37%2,294 outcomes observed | 742,294 trajectories | Final |
| 12 | minimax-m3.1-flash | 16.39%14.93%–17.96% | 75.50% (1,732 / 2,294) · limit-constrained | 376 / 2,294 | 100.00% | 24.41%2,294 outcomes observed | 302,294 trajectories | Final |
| 1 | deepseek-v4.1-flash-1017 | 79.40%75.64%–82.71% | 2.40% (12 / 500) | 397 / 500 | 100.00% | 97.60%500 outcomes observed | 16500 trajectories | Final |
|---|---|---|---|---|---|---|---|---|
| 2 | glm-5.3 | 78.80%75.00%–82.16% | 1.80% (9 / 500) | 394 / 500 | 100.00% | 98.20%500 outcomes observed | 26500 trajectories | Final |
| 3 | deepseek-v4.1-flash-1017-uncensored-fp8 | 77.00%73.11%–80.47% | 2.60% (13 / 500) | 385 / 500 | 100.00% | 97.40%500 outcomes observed | 16500 trajectories | Final |
| 4 | glm-5.3-flash-uncensored-fp8 | 76.20%72.28%–79.72% | 3.40% (17 / 500) | 381 / 500 | 100.00% | 96.60%500 outcomes observed | 22500 trajectories | Final |
| 5 | glm-5.3-flash | 74.00%69.98%–77.65% | 7.20% (36 / 500) | 370 / 500 | 100.00% | 92.80%500 outcomes observed | 23500 trajectories | Final |
| 6 | kimi-k3 | 72.60%68.53%–76.33% | 14.20% (71 / 500) | 363 / 500 | 100.00% | 85.80%500 outcomes observed | 29500 trajectories | Final |
| 7 | nemotron-3-ultra | 68.00%63.79%–71.94% | 2.20% (11 / 500) | 340 / 500 | 100.00% | 97.00%500 outcomes observed | 57.5500 trajectories | Final |
| 8 | kimi-k3-abliterated | 62.20%57.87%–66.34% | 30.80% (154 / 500) · limit-constrained | 311 / 500 | 100.00% | 69.20%500 outcomes observed | 28500 trajectories | Final |
| 9 | nemotron-3-super | 56.80%52.42%–61.07% | 4.80% (24 / 500) | 284 / 500 | 100.00% | 95.00%500 outcomes observed | 47.5500 trajectories | Final |
| 10 | glm-5.3-uncensored-fp8 | 56.20%51.82%–60.49% | 33.20% (166 / 500) · limit-constrained | 281 / 500 | 100.00% | 66.40%500 outcomes observed | 30500 trajectories | Final |
| 11 | nemotron-3.5-lightning | 29.20%25.39%–33.33% | 8.20% (41 / 500) | 146 / 500 | 100.00% | 91.80%500 outcomes observed | 70500 trajectories | Final |
| 12 | minimax-m3.1-flash | 28.60%24.81%–32.71% | 66.20% (331 / 500) · limit-constrained | 143 / 500 | 100.00% | 33.60%500 outcomes observed | 31500 trajectories | Final |
| 1 | deepseek-v4.1-flash-1017 | 66.33%60.81%–71.44% | 2.00% (6 / 300) | 199 / 300 | 100.00% | 97.67%300 outcomes observed | 16300 trajectories | Final |
|---|---|---|---|---|---|---|---|---|
| 1 | glm-5.3 | 66.33%60.81%–71.44% | 2.00% (6 / 300) | 199 / 300 | 100.00% | 98.00%300 outcomes observed | 28300 trajectories | Final |
| 3 | deepseek-v4.1-flash-1017-uncensored-fp8 | 65.67%60.13%–70.81% | 1.67% (5 / 300) | 197 / 300 | 100.00% | 98.33%300 outcomes observed | 17300 trajectories | Final |
| 4 | glm-5.3-flash | 61.67%56.05%–66.99% | 7.33% (22 / 300) | 185 / 300 | 100.00% | 92.67%300 outcomes observed | 24300 trajectories | Final |
| 4 | glm-5.3-flash-uncensored-fp8 | 61.67%56.05%–66.99% | 5.67% (17 / 300) | 185 / 300 | 100.00% | 94.33%300 outcomes observed | 23.5300 trajectories | Final |
| 6 | kimi-k3 | 60.67%55.04%–66.03% | 17.00% (51 / 300) | 182 / 300 | 100.00% | 83.00%300 outcomes observed | 30.5300 trajectories | Final |
| 7 | nemotron-3-ultra | 53.67%48.01%–59.23% | 1.67% (5 / 300) | 161 / 300 | 100.00% | 97.00%300 outcomes observed | 59300 trajectories | Final |
| 8 | kimi-k3-abliterated | 49.67%44.05%–55.29% | 36.67% (110 / 300) · limit-constrained | 149 / 300 | 100.00% | 63.33%300 outcomes observed | 26300 trajectories | Final |
| 9 | glm-5.3-uncensored-fp8 | 47.33%41.75%–52.98% | 36.00% (108 / 300) · limit-constrained | 142 / 300 | 100.00% | 63.33%300 outcomes observed | 30300 trajectories | Final |
| 10 | nemotron-3-super | 44.00%38.49%–49.66% | 6.00% (18 / 300) | 132 / 300 | 100.00% | 93.67%300 outcomes observed | 50300 trajectories | Final |
| 11 | minimax-m3.1-flash | 22.33%17.99%–27.38% | 69.33% (208 / 300) · limit-constrained | 67 / 300 | 100.00% | 30.33%300 outcomes observed | 31300 trajectories | Final |
| 11 | nemotron-3.5-lightning | 22.33%17.99%–27.38% | 5.33% (16 / 300) | 67 / 300 | 100.00% | 94.67%300 outcomes observed | 70300 trajectories | Final |
Ranks follow the displayed score; ties share rank. Submit rate is Submitted exits / observed generation outcomes, including empty submissions. Steps are recorded model API calls. Missing usage is excluded, never counted as zero. Intervals do not capture gateway, scaffold or repeated-run variability.
Our results beside the official leaderboard
Where our 12 models land among the 24 official entries for the full test set
Each grey dot is one entry on the public SWE-bench leaderboard. Our models are the coloured dots below them. Scores show the share of 2,294 tasks solved. Our scores run from 16.4% to 55.0%; the official entries run from 0.2% to 52.6%, with a middle value of 18.5%.
Showing all 24 official entries, with any agent, model or number of attempts.
Entry from another teamEntry that used mini-SWE-agent, the same simple agent we usedEntry that used an open-weight model (diamond)
Keyboard: Tab to a row, arrow keys to move between dots, Escape to close the card.
Numbers match the model key. Nearby model markers are spaced apart for readability; exact scores are below. The light band shows where official entries cluster.
Official entries use different agents, budgets, tools and attempt counts, and they are not run by us. This is context, not a controlled comparison or an official submission.
Snapshot of swebench.com retrieved 2026-10-29 (commit 193160a, sha256 83cd949a). No official entry shares an exact model name with ours.
Show the official entries as a table
| Name | Agent | Model | Date | Attempts | |
|---|---|---|---|---|---|
| Sonar Foundation Agent + Claude 4.5 Opus | Sonar Foundation Agent | Claude 4.5 Opus | 52.62% | 2025-12-19 | 1 |
| Salesforce AI Research SAGE (bash-only) | Salesforce AI Research SAGE | Multiple | 44.25% | 2025-10-27 | 2+ |
| Atlassian Rovo Dev (2025-06-05) | Atlassian Rovo Dev | Undisclosed | 41.98% | 2025-06-05 | 1 |
| Amazon Q Developer Agent (v20250405-dev) | Amazon Q Developer Agent | Undisclosed | 37.10% | 2025-05-22 | 2+ |
| SWE-agent 1.0 (Claude 3.7 Sonnet) | SWE-agent 1.0 | Claude 3.7 Sonnet | 33.83% | 2025-02-27 | 1 |
| Amazon Q Developer Agent (v20241202-dev) | Amazon Q Developer Agent | Undisclosed | 29.99% | 2025-01-31 | 1 |
| OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) | OpenHands | CodeAct v2.1 (claude-3-5-sonnet-20241022) | 29.38% | 2024-11-03 | 1 |
| AutoCodeRover-v2.0 (Claude-3.5-Sonnet-20241022) | AutoCodeRover-v2.0 | Claude 3.5-Sonnet-20241022 | 24.89% | 2024-11-21 | 1 |
| Honeycomb | Honeycomb | Undisclosed | 22.06% | 2024-08-20 | 1 |
| Amazon Q Developer Agent (v20240719-dev) | Amazon Q Developer Agent | Undisclosed | 19.75% | 2024-07-21 | 1 |
| Factory Code Droid | Factory Code Droid | Undisclosed | 19.27% | 2024-06-17 | 1 |
| AutoCodeRover (v20240620) + GPT 4o (2024-05-13) | AutoCodeRover | GPT-4o | 18.83% | 2024-06-28 | 1 |
| SWE-agent + Claude 3.5 Sonnet | SWE-agent | Claude 3.5 Sonnet | 18.13% | 2024-06-20 | 1 |
| AppMap Navie + GPT 4o (2024-05-13) | AppMap Navie | GPT-4o | 14.60% | 2024-06-15 | 1 |
| Amazon Q Developer Agent (v20240430-dev) | Amazon Q Developer Agent | Undisclosed | 13.82% | 2024-05-09 | 1 |
| SWE-agent + GPT 4 (1106) | SWE-agent | GPT-4 (1106) | 12.47% | 2024-04-02 | 1 |
| SWE-agent + GPT 4o (2024-05-13) | SWE-agent | GPT-4o | 11.99% | 2024-07-28 | 1 |
| SWE-agent + Claude 3 Opus | SWE-agent | Claude 3 Opus | 10.51% | 2024-04-02 | 1 |
| RAG + Claude 3 Opus | RAG baseline | Claude 3 Opus | 3.79% | 2024-04-02 | 1 |
| RAG + Claude 2 | RAG baseline | Claude 2 | 1.96% | 2023-10-10 | 1 |
| RAG + GPT 4 (1106) | RAG baseline | GPT-4 (1106) | 1.31% | 2024-04-02 | 1 |
| RAG + SWE-Llama 13B | RAG baseline | SWE-Llama 13B | 0.70% | 2023-10-10 | 1 |
| RAG + SWE-Llama 7B | RAG baseline | SWE-Llama 7B | 0.70% | 2023-10-10 | 1 |
| RAG + ChatGPT 3.5 | RAG baseline | GPT-3.5 | 0.17% | 2023-10-10 | 1 |
| 0 of 24 plotted entries score higher than our highest result; 11 of 24 score lower than our lowest result. | |||||
| Name | Agent | Model | Date | Attempts | |
|---|---|---|---|---|---|
| Sonar Foundation Agent + Claude 4.5 Opus | Sonar Foundation Agent | Claude 4.5 Opus | 79.20% | 2025-12-05 | 1 |
| live-SWE-agent + Claude 4.5 Opus medium (20251101) | live-SWE-agent | Claude 4.5 Opus | 79.20% | 2025-12-15 | 1 |
| TRAE + Doubao-Seed-Code | TRAE | Doubao-Seed-Code | 78.80% | 2025-09-28 | 2+ |
| live-SWE-agent + Gemini 3 Pro Preview (2025-11-18) | live-SWE-agent | Gemini 3 Pro Preview | 77.40% | 2025-11-20 | 1 |
| EPAM AI/Run Developer Agent v20250719 + Claude 4 Sonnet | EPAM AI/Run Developer Agent | Claude 4 Sonnet | 76.80% | 2025-08-04 | 2+ |
| Atlassian Rovo Dev (2025-09-02) | Atlassian Rovo Dev | Multiple | 76.80% | 2025-09-02 | 2+ |
| Claude 4.5 Opus (high) | mini-SWE-agent | Claude 4.5 Opus | 76.80% | 2026-02-17 | 1 |
| ACoder | ACoder | Multiple | 76.40% | 2025-08-19 | 2+ |
| Gemini 3 Flash (high) | mini-SWE-agent | Gemini 3 Flash | 75.80% | 2026-02-17 | 1 |
| MiniMax M2.5 (high) | mini-SWE-agent | MiniMax M2.5 | 75.80% | 2026-02-17 | 1 |
| Warp | Warp | Multiple | 75.60% | 2025-09-01 | 2+ |
| Claude 4.6 Opus | mini-SWE-agent | Claude 4.6 Opus | 75.60% | 2026-02-17 | 1 |
| TRAE + Claude Sonnet 4 + Opus 4 + Sonnet 3.7 + Gemini 2.5 Pro | TRAE | Multiple | 75.20% | 2025-06-12 | 2+ |
| Harness AI | Harness AI | Claude Sonnet 4 | 74.80% | 2025-07-31 | 1 |
| Sonar Foundation Agent + Claude 4.5 Sonnet | Sonar Foundation Agent | Claude 4.5 Sonnet | 74.80% | 2025-11-03 | 1 |
| Lingxi-v1.5_claude-4-sonnet-20250514 | Lingxi-v1.5 | Claude 4 Sonnet | 74.60% | 2025-07-20 | 1 |
| JoyCode + Claude 4 Sonnet + GPT-4.1 | JoyCode | Multiple | 74.60% | 2025-09-15 | 1 |
| Refact.ai Agent + Claude 4 Sonnet + o4-mini | Refact.ai Agent | Multiple | 74.40% | 2025-06-03 | 1 |
| Prometheus-v1.2.1 + GPT-5 | Prometheus-v1.2.1 | GPT-5 | 74.40% | 2025-10-15 | 2+ |
| Claude 4.5 Opus (20251101) (medium) | mini-SWE-agent | Claude 4.5 Opus | 74.40% | 2025-11-24 | 1 |
| Gemini 3 Pro Preview (2025-11-18) | mini-SWE-agent | Gemini 3 Pro Preview | 74.20% | 2025-11-18 | 1 |
| Salesforce AI Research SAGE (OpenHands) | Salesforce AI Research SAGE | Multiple | 73.80% | 2025-11-03 | 2+ |
| Tools + Claude 4 Opus (2025-05-22) | Tools | Claude 4 Opus | 73.20% | 2025-05-22 | 1 |
| Salesforce AI Research SAGE (bash-only) | Salesforce AI Research SAGE | Multiple | 73.00% | 2025-10-21 | 2+ |
| GLM 5 (high) | mini-SWE-agent | GLM 5 | 72.80% | 2026-02-17 | 1 |
| GPT 5.2 (high) | mini-SWE-agent | GPT 5.2 | 72.80% | 2026-02-17 | 1 |
| GPT 5.2 Codex | mini-SWE-agent | GPT 5.2 Codex | 72.80% | 2026-02-19 | 1 |
| Tools + Claude 4 Sonnet (2025-05-22) | Tools | Claude 4 Sonnet | 72.40% | 2025-05-22 | 1 |
| OpenHands + GPT-5 | OpenHands | GPT-5 | 71.80% | 2025-08-07 | 1 |
| GPT 5.2 (2025-12-11) (high) | mini-SWE-agent | GPT 5.2 | 71.80% | 2025-12-11 | 1 |
| Claude 4.5 Sonnet (high) | mini-SWE-agent | Claude 4.5 Sonnet | 71.40% | 2026-02-17 | 1 |
| Bloop | Bloop | Claude 4 Sonnet | 71.20% | 2025-07-10 | 1 |
| Qodo Command | Qodo Command | Multiple | 71.20% | 2025-07-15 | 1 |
| Prometheus-v1.2 + GPT-5 | Prometheus-v1.2 | GPT-5 | 71.20% | 2025-09-29 | 1 |
| Lingxi v1.5 x Kimi K2 | Lingxi v1.5 | Kimi K2 | 71.20% | 2025-10-14 | 1 |
| Warp | Warp | Multiple | 71.00% | 2025-06-23 | 1 |
| Moatless Tools + Claude 4 Sonnet | Moatless Tools | Claude 4 Sonnet | 70.80% | 2025-06-11 | 1 |
| Kimi K2.5 (high) | mini-SWE-agent | Kimi K2.5 | 70.80% | 2026-02-17 | 1 |
| TRAE | TRAE | Undisclosed | 70.60% | 2025-05-19 | 2+ |
| Claude 4.5 Sonnet (20250929) | mini-SWE-agent | Claude 4.5 Sonnet | 70.60% | 2025-09-29 | 1 |
| Refact.ai Agent | Refact.ai Agent | Multiple | 70.40% | 2025-05-15 | 1 |
| OpenHands + Claude 4 Sonnet | OpenHands | Claude 4 Sonnet | 70.40% | 2025-05-24 | 1 |
| Augment Agent v1 | Augment Agent v1 | Undisclosed | 70.40% | 2025-06-10 | 1 |
| devlo | devlo | Multiple | 70.20% | 2025-05-19 | 1 |
| Zencoder (2025-04-30) | Zencoder | Multiple | 70.00% | 2025-04-30 | 2+ |
| DeepSeek V3.2 (high) | mini-SWE-agent | DeepSeek V3.2 | 70.00% | 2026-02-17 | 1 |
| OpenHands + Qwen3-Coder-480B-A35B-Instruct | OpenHands | Qwen3-Coder-480B-A35B-Instruct | 69.60% | 2025-08-05 | 1 |
| Gemini 3 Pro | mini-SWE-agent | Gemini 3 Pro | 69.60% | 2026-02-26 | 1 |
| GPT 5.2 (2025-12-11) | mini-SWE-agent | GPT 5.2 | 69.00% | 2025-12-11 | 1 |
| Nemotron-CORTEXA | Nemotron-CORTEXA | Multiple | 68.20% | 2025-05-16 | 2+ |
| GLM-4.6 | Undisclosed | GLM-4.6 | 68.20% | 2025-09-30 | 1 |
| Claude 4 Opus (20250514) | mini-SWE-agent | Claude 4 Opus | 67.60% | 2025-08-02 | 1 |
| SWE-agent + Claude 4 Sonnet | SWE-agent | Claude 4 Sonnet | 66.60% | 2025-05-22 | 1 |
| Claude 4.5 Haiku (high) | mini-SWE-agent | Claude 4.5 Haiku | 66.60% | 2026-02-17 | 1 |
| Aime-coder v1 + Anthopic Claude 3.7 Sonnet | Aime-coder v1 | Claude 3.7 Sonnet | 66.40% | 2025-05-14 | 2+ |
| GPT 5.1 (2025-11-13) (medium) | mini-SWE-agent | GPT 5.1 | 66.00% | 2025-11-20 | 1 |
| GPT 5.1 Codex (medium) | mini-SWE-agent | GPT 5.1 Codex | 66.00% | 2025-11-24 | 1 |
| OpenHands | OpenHands | Undisclosed | 65.80% | 2025-04-15 | 2+ |
| Augment Agent v0 | Augment Agent v0 | Undisclosed | 65.40% | 2025-03-16 | 2+ |
| Amazon Q Developer Agent (v20250405-dev) | Amazon Q Developer Agent | Undisclosed | 65.40% | 2025-04-05 | 2+ |
| OpenHands + Kimi K2 | OpenHands | Kimi K2 | 65.40% | 2025-07-16 | 1 |
| GPT 5 (2025-08-07) (medium) | mini-SWE-agent | GPT 5 | 65.00% | 2025-08-07 | 1 |
| Claude 4 Sonnet (20250514) | mini-SWE-agent | Claude 4 Sonnet | 64.93% | 2025-07-26 | 1 |
| W&B Programmer O1 crosscheck5 | W&B Programmer O1 crosscheck5 | o1-preview | 64.60% | 2025-01-17 | 2+ |
| PatchPilot-v1.1 | PatchPilot-v1.1 | o4-mini | 64.60% | 2025-05-03 | 2+ |
| GLM-4.5 | Undisclosed | GLM-4.5 | 64.20% | 2025-07-28 | 1 |
| AgentScope | AgentScope | Claude 3.5 Sonnet | 63.40% | 2025-02-06 | 2+ |
| Kimi K2 Thinking | mini-SWE-agent | Kimi K2 Thinking | 63.40% | 2025-12-10 | 1 |
| Tools + Claude 3.7 Sonnet (2025-02-24) | Tools | Claude 3.7 Sonnet | 63.20% | 2025-02-24 | 2+ |
| Blackbox AI Agent | Blackbox AI Agent | Undisclosed | 62.80% | 2025-01-10 | 2+ |
| EPAM AI/Run Developer Agent v20250219 + Anthopic Claude 3.5 Sonnet | EPAM AI/Run Developer Agent | Claude 3.5 Sonnet | 62.80% | 2025-02-28 | 2+ |
| SWE-agent + Claude 3.7 Sonnet w/ Review Heavy | SWE-agent | Claude 3.7 Sonnet w/ Review Heavy | 62.40% | 2025-02-25 | 2+ |
| CodeStory Midwit Agent + swe-search | CodeStory Midwit Agent | swe-search | 62.20% | 2024-12-21 | 2+ |
| MiniMax M2 | mini-SWE-agent | MiniMax M2 | 61.00% | 2025-11-24 | 1 |
| OpenHands + 4x Scaled (2024-02-03) | OpenHands | 4x Scaled | 60.80% | 2025-02-03 | 2+ |
| EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct | EntroPO + R2E | Qwen3-Coder-30B-A3B-Instruct | 60.40% | 2025-09-01 | 2+ |
| Learn-by-interact | Learn-by-interact | Claude 3.5 Sonnet | 60.20% | 2025-01-10 | 2+ |
| DeepSeek V3.2 Reasoner | mini-SWE-agent | DeepSeek V3.2 Reasoner | 60.00% | 2025-12-01 | 1 |
| GPT 5 mini (2025-08-07) (medium) | mini-SWE-agent | GPT 5 mini | 59.80% | 2025-08-07 | 1 |
| DeepSWE-Preview + TTS(Bo16) | DeepSWE-Preview | TTS(Bo16) | 58.80% | 2025-06-29 | 2+ |
| o3 (2025-04-16) | mini-SWE-agent | o3 | 58.40% | 2025-07-26 | 1 |
| devlo | devlo | Undisclosed | 58.20% | 2024-12-13 | 2+ |
| Nemotron-CORTEXA | Nemotron-CORTEXA | Multiple | 58.20% | 2025-04-10 | 2+ |
| Emergent E1 (v2024-12-23) | Emergent E1 | Multiple | 57.20% | 2024-12-23 | 2+ |
| Gru(2024-12-08) | Gru | Undisclosed | 57.00% | 2024-12-08 | 2+ |
| Artemis Agent v2 (2025-09-24) | Artemis Agent v2 | Claude Sonnet 4 | 57.00% | 2025-09-24 | 1 |
| SWE-Rizzo | SWE-Rizzo | Undisclosed | 56.60% | 2025-04-05 | 2+ |
| Devstral Small (2512) | mini-SWE-agent | Devstral Small (2512) | 56.40% | 2025-12-09 | 1 |
| GPT 5 mini | mini-SWE-agent | GPT 5 mini | 56.20% | 2026-02-17 | 1 |
| EPAM AI/Run Developer Agent v20241212 + Anthopic Claude 3.5 Sonnet | EPAM AI/Run Developer Agent | Claude 3.5 Sonnet | 55.40% | 2024-12-12 | 2+ |
| Qwen3-Coder 480B/A35B Instruct | mini-SWE-agent | Qwen3-Coder 480B/A35B Instruct | 55.40% | 2025-08-02 | 1 |
| GLM 4.6 (T=1) | mini-SWE-agent | GLM 4.6 | 55.40% | 2025-12-01 | 1 |
| Amazon Q Developer Agent (v20241202-dev) | Amazon Q Developer Agent | Undisclosed | 55.00% | 2024-12-02 | 2+ |
| devlo | devlo | Claude 3.5 Sonnet | 54.20% | 2024-11-08 | 2+ |
| GLM 4.5 (2025-08-22) | mini-SWE-agent | GLM 4.5 | 54.20% | 2025-08-22 | 1 |
| Devstral (2512) | mini-SWE-agent | Devstral (2512) | 53.80% | 2025-12-09 | 1 |
| Gemini 2.5 Pro (2025-05-06) | mini-SWE-agent | Gemini 2.5 Pro | 53.60% | 2025-07-26 | 1 |
| FrogBoss-32B-2510 | FrogBoss-32B-2510 | Frogboss 32B 2510 | 53.60% | 2025-11-10 | 1 |
| CodeSweep - SWE-agent - Kimi K2 Instruct | CodeSweep - SWE-agent | Kimi K2 Instruct | 53.40% | 2025-08-04 | 1 |
| Bracket.sh | Bracket.sh | Undisclosed | 53.20% | 2025-01-20 | 2+ |
| OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) | OpenHands | CodeAct v2.1 (claude-3-5-sonnet-20241022) | 53.00% | 2024-10-29 | 2+ |
| Claude 3.7 Sonnet (20250219) | mini-SWE-agent | Claude 3.7 Sonnet | 52.80% | 2025-07-20 | 1 |
| Google Jules + Gemini 2.0 Flash (v20241212-experimental) | Google Jules | Gemini 2.0 Flash (v20241212-experimental) | 52.20% | 2024-12-12 | 2+ |
| EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct | EntroPO + R2E | Qwen3-Coder-30B-A3B-Instruct | 52.20% | 2025-09-01 | 1 |
| Engine Labs (2024-11-25) | Engine Labs | Claude 3.5 Sonnet | 51.80% | 2024-11-25 | 1 |
| AutoCodeRover-v2.1 (Claude-3.5-Sonnet-20241022) | AutoCodeRover-v2.1 | Claude 3.5-Sonnet-20241022 | 51.60% | 2025-01-22 | 2+ |
| OpenHands + Qwen3-Coder-30B-A3B-Instruct | OpenHands | Qwen3-Coder-30B-A3B-Instruct | 51.60% | 2025-08-05 | 1 |
| Agentless-1.5 + Claude-3.5 Sonnet (20241022) | Agentless-1.5 | Claude 3.5 Sonnet | 50.80% | 2024-12-02 | 2+ |
| Solver (2024-10-28) | Solver | Undisclosed | 50.00% | 2024-10-28 | 2+ |
| Bytedance MarsCode Agent | Bytedance MarsCode Agent | Undisclosed | 50.00% | 2024-11-25 | 2+ |
| nFactorial (2024-11-05) | nFactorial | Multiple | 49.20% | 2024-11-05 | 2+ |
| Tools + Claude 3.5 Sonnet (2024-10-22) | Tools | Claude 3.5 Sonnet | 49.00% | 2024-10-22 | 2+ |
| Composio SWE-Kit (2024-10-25) | Composio SWE-Kit | Multiple | 48.60% | 2024-10-25 | 2+ |
| AppMap Navie v2 | AppMap Navie v2 | Multiple | 47.20% | 2024-11-06 | 2+ |
| Skywork-SWE-32B + TTS(Bo8) | Skywork-SWE-32B | TTS(Bo8) | 47.00% | 2025-06-16 | 2+ |
| OpenHands + DevStral Small 2505 | OpenHands | DevStral Small 2505 | 46.80% | 2025-05-20 | 1 |
| Emergent E1 (v2024-10-12) | Emergent E1 | Multiple | 46.60% | 2024-10-23 | 2+ |
| AutoCodeRover-v2.0 (Claude-3.5-Sonnet-20241022) | AutoCodeRover-v2.0 | Claude 3.5-Sonnet-20241022 | 46.20% | 2024-11-08 | 2+ |
| PatchPilot + Co-PatcheR | PatchPilot | Co-PatcheR | 46.00% | 2025-05-28 | 2+ |
| Solver (2024-09-12) | Solver | Undisclosed | 45.40% | 2024-09-24 | 2+ |
| Gru(2024-08-24) | Gru | Undisclosed | 45.20% | 2024-08-24 | 2+ |
| o4-mini (2025-04-16) | mini-SWE-agent | o4-mini | 45.00% | 2025-07-26 | 1 |
| FrogMini-14B-2510 | FrogMini-14B-2510 | Frogmini 14B 2510 | 45.00% | 2025-11-10 | 1 |
| CodeShellAgent + Gemini 2.0 Flash (Experimental) | CodeShellAgent | Gemini 2.0 Flash (Experimental) | 44.20% | 2025-01-18 | 2+ |
| Kimi K2 Instruct | mini-SWE-agent | Kimi K2 Instruct | 43.80% | 2025-08-07 | 1 |
| Solver (2024-09-12) | Solver | Undisclosed | 43.60% | 2024-09-20 | 2+ |
| Agentless Lite + O3 Mini (20250214) | Agentless Lite | O3 Mini | 42.40% | 2025-02-14 | 2+ |
| Amazon Nova Premier 1.0 (2025-04-30) | Amazon Nova Premier 1.0 | Amazon.nova Premier v1:0 | 42.40% | 2025-05-27 | 2+ |
| DeepSWE-Preview | R2E-Gym | DeepSWE-Preview | 42.20% | 2025-06-29 | 1 |
| SWE-Exp | SWE-Exp | DeepSeek V3 0324 | 42.00% | 2025-08-06 | 2+ |
| nFactorial (2024-10-30) | nFactorial | Multiple | 41.60% | 2024-10-30 | 2+ |
| ugaiforge | ugaiforge | Undisclosed | 41.60% | 2025-01-12 | 2+ |
| SWE-RL (Llama3-SWE-RL-70B + Agentless Mini) (20250226) | Agentless Mini | Llama3-SWE-RL-70B | 41.20% | 2025-02-26 | 2+ |
| Honeycomb | Honeycomb | Undisclosed | 40.60% | 2024-08-20 | 2+ |
| Composio SWEkit + Claude 3.5 Sonnet (2024-10-16) | Composio SWEkit | Claude 3.5 Sonnet | 40.60% | 2024-10-16 | 2+ |
| Tools + Claude 3.5 Haiku | Tools | Claude 3.5 Haiku | 40.60% | 2024-10-22 | 2+ |
| Nebius AI Qwen 2.5 72B Generator + LLama 3.1 70B Critic | Nebius AI | Multiple | 40.60% | 2024-11-13 | 2+ |
| SWE-agent + SWE-agent-LM-32B | SWE-agent | SWE-agent-LM-32B | 40.20% | 2025-05-11 | 1 |
| EPAM AI/Run Developer Agent v20241029 + Anthopic Claude 3.5 Sonnet | EPAM AI/Run Developer Agent | Claude 3.5 Sonnet | 39.60% | 2024-10-29 | 2+ |
| GPT 4.1 (2025-04-14) | mini-SWE-agent | GPT 4.1 | 39.58% | 2025-07-26 | 1 |
| Amazon Q Developer Agent (v20240719-dev) | Amazon Q Developer Agent | Undisclosed | 38.80% | 2024-07-21 | 2+ |
| Agentless-1.5 + GPT 4o (2024-05-13) | Agentless-1.5 | GPT-4o | 38.80% | 2024-10-28 | 1 |
| AutoCodeRover (v20240620) + GPT 4o (2024-05-13) | AutoCodeRover | GPT-4o | 38.40% | 2024-06-28 | 2+ |
| Skywork-SWE-32B | Skywork-SWE-32B | Qwen2.5 Coder 32B Instruct | 38.00% | 2025-06-16 | 1 |
| SWE-agent + DevStral Small 2507 | SWE-agent | DevStral Small 2507 | 38.00% | 2025-07-25 | 1 |
| Factory Code Droid | Factory Code Droid | Undisclosed | 37.00% | 2024-06-17 | 2+ |
| GPT 5 nano (2025-08-07) (medium) | mini-SWE-agent | GPT 5 nano | 34.80% | 2025-08-07 | 1 |
| SWE-agent + Claude 3.5 Sonnet | SWE-agent | Claude 3.5 Sonnet | 33.60% | 2024-06-20 | 1 |
| SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) | SWE-Fixer | Qwen2.5 (7B + 72B) | 32.80% | 2025-03-06 | 1 |
| MASAI + GPT 4o (2024-06-12) | MASAI | GPT-4o | 32.60% | 2024-06-12 | 2+ |
| Artemis Agent v1 (2024-11-20) | Artemis Agent v1 | Undisclosed | 32.00% | 2024-11-20 | 2+ |
| nFactorial (2024-10-07) | nFactorial | Undisclosed | 31.60% | 2024-10-07 | 2+ |
| SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) 20241128 | SWE-Fixer | Qwen2.5 (7B + 72B) | 30.20% | 2024-11-28 | 2+ |
| Lingma Agent + Lingma SWE-GPT 72b (v0925) | Lingma Agent | Lingma SWE-GPT 72b (v0925) | 28.80% | 2024-10-02 | 2+ |
| Gemini 2.5 Flash (2025-04-17) | mini-SWE-agent | Gemini 2.5 Flash | 28.73% | 2025-07-26 | 1 |
| EPAM AI/Run Developer Agent + GPT4o | EPAM AI/Run Developer Agent | GPT-4o | 27.00% | 2024-10-16 | 2+ |
| AppMap Navie + GPT 4o (2024-05-13) | AppMap Navie | GPT-4o | 26.20% | 2024-06-15 | 2+ |
| gpt-oss-120b | mini-SWE-agent | gpt-oss-120b | 26.00% | 2025-08-07 | 1 |
| nFactorial (2024-10-01) | nFactorial | Undisclosed | 25.80% | 2024-10-01 | 2+ |
| Amazon Q Developer Agent (v20240430-dev) | Amazon Q Developer Agent | Undisclosed | 25.60% | 2024-05-09 | 2+ |
| Lingma Agent + Lingma SWE-GPT 72b (v0918) | Lingma Agent | Lingma SWE-GPT 72b (v0918) | 25.00% | 2024-09-18 | 2+ |
| EPAM AI/Run Developer Agent + GPT4o | EPAM AI/Run Developer Agent | GPT-4o | 24.00% | 2024-08-20 | 2+ |
| GPT 4.1 mini (2025-04-14) | mini-SWE-agent | GPT 4.1 mini | 23.94% | 2025-07-20 | 1 |
| SWE-agent + GPT 4o (2024-05-13) | SWE-agent | GPT-4o | 23.20% | 2024-07-28 | 1 |
| MCTS-Refine-7B | MCTS-Refine-7B | MCTS Refine 7B | 23.20% | 2025-06-27 | 1 |
| SWE-agent + GPT 4 (1106) | SWE-agent | GPT-4 (1106) | 22.40% | 2024-04-02 | 1 |
| GPT 4o (2024-11-20) | mini-SWE-agent | GPT 4o | 21.62% | 2025-07-20 | 1 |
| Llama 4 Maverick Instruct | mini-SWE-agent | Llama 4 Maverick Instruct | 21.04% | 2025-07-20 | 1 |
| Lingma Agent + Lingma SWE-GPT 7b (v0925) | Lingma Agent | Lingma SWE-GPT 7b (v0925) | 18.20% | 2024-10-02 | 2+ |
| SWE-agent + Claude 3 Opus | SWE-agent | Claude 3 Opus | 15.80% | 2024-04-02 | 1 |
| Gemini 2.0 Flash | mini-SWE-agent | Gemini 2.0 Flash | 13.52% | 2025-07-26 | 1 |
| Lingma Agent + Lingma SWE-GPT 7b (v0918) | Lingma Agent | Lingma SWE-GPT 7b (v0918) | 10.20% | 2024-09-18 | 2+ |
| Llama 4 Scout Instruct | mini-SWE-agent | Llama 4 Scout Instruct | 9.06% | 2025-07-20 | 1 |
| Qwen2.5-Coder 32B Instruct | mini-SWE-agent | Qwen2.5-Coder 32B Instruct | 9.00% | 2025-08-03 | 1 |
| RAG + Claude 3 Opus | RAG baseline | Claude 3 Opus | 7.00% | 2024-04-02 | 2+ |
| RAG + Claude 2 | RAG baseline | Claude 2 | 4.40% | 2023-10-10 | 1 |
| RAG + GPT 4 (1106) | RAG baseline | GPT-4 (1106) | 2.80% | 2024-04-02 | 2+ |
| RAG + SWE-Llama 7B | RAG baseline | SWE-Llama 7B | 1.40% | 2023-10-10 | 1 |
| RAG + SWE-Llama 13B | RAG baseline | SWE-Llama 13B | 1.20% | 2023-10-10 | 1 |
| RAG + ChatGPT 3.5 | RAG baseline | GPT-3.5 | 0.40% | 2023-10-10 | 1 |
| 0 of 180 plotted entries score higher than our highest result; 25 of 180 score lower than our lowest result. | |||||
| Name | Agent | Model | Date | Attempts | |
|---|---|---|---|---|---|
| ExpeRepair-v1.0 + Claude 4 Sonnet | ExpeRepair-v1.0 | Claude 4 Sonnet | 60.33% | 2025-06-25 | 2+ |
| Refact.ai Agent | Refact.ai Agent | Multiple | 60.00% | 2025-04-25 | 1 |
| KGCompass + Claude 4 Sonnet (20250514) | KGCompass | Claude 4 Sonnet | 58.33% | 2025-09-06 | 2+ |
| SWE-agent + Claude 4 Sonnet | SWE-agent | Claude 4 Sonnet | 56.67% | 2025-05-26 | 1 |
| Isoform | Isoform | Undisclosed | 55.00% | 2025-01-14 | 2+ |
| SemAgent_Multi-v1.0 | SemAgent_Multi-v1.0 | Multiple | 51.67% | 2025-06-25 | 2+ |
| Isea | Isea | Claude 3.5 Sonnet | 51.33% | 2025-09-11 | 2+ |
| EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct | EntroPO + R2E | Qwen3-Coder-30B-A3B-Instruct | 49.67% | 2025-09-01 | 2+ |
| Blackbox AI Agent | Blackbox AI Agent | Undisclosed | 49.00% | 2024-12-20 | unknown |
| Codev | Codev | Multiple | 49.00% | 2025-05-28 | 1 |
| Gru(2024-12-08) | Gru | Undisclosed | 48.67% | 2024-12-08 | unknown |
| Globant Code Fixer Agent | Globant Code Fixer Agent | Undisclosed | 48.33% | 2024-11-27 | unknown |
| ExpeRepair-v1.0 | ExpeRepair-v1.0 | Multiple | 48.33% | 2025-06-13 | 2+ |
| SWE-agent + Claude 3.7 Sonnet | SWE-agent | Claude 3.7 Sonnet | 48.00% | 2025-02-26 | 1 |
| devlo | devlo | Claude 3.5 Sonnet | 47.33% | 2024-11-22 | unknown |
| DARS Agent | DARS Agent | Multiple | 47.00% | 2025-02-05 | unknown |
| KGCompass + Claude 3.5 Sonnet (20241022) | KGCompass | Claude 3.5 Sonnet | 46.00% | 2025-06-19 | 2+ |
| EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct | EntroPO + R2E | Qwen3-Coder-30B-A3B-Instruct | 45.00% | 2025-09-01 | 1 |
| Kodu-v1 + Claude-3.5 Sonnet (20241022) | Kodu-v1 | Claude 3.5 Sonnet | 44.67% | 2024-12-07 | unknown |
| CodeFuse-CGM | CodeFuse-CGM | Undisclosed | 44.00% | 2025-03-10 | 2+ |
| CodeStory Aide + Mixed Models | CodeStory Aide | Mixed Models | 43.00% | 2024-07-02 | unknown |
| Lingxi | Lingxi | Claude 3.5 Sonnet | 42.67% | 2025-05-09 | unknown |
| OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) | OpenHands | CodeAct v2.1 (claude-3-5-sonnet-20241022) | 41.67% | 2024-10-25 | unknown |
| Codart AI | Codart AI | Multiple | 41.67% | 2025-05-15 | 1 |
| PatchKitty-0.9 + Claude-3.5 Sonnet (20241022) | PatchKitty-0.9 | Claude 3.5 Sonnet | 41.33% | 2024-12-20 | unknown |
| Composio SWE-Kit (2024-10-30) | Composio SWE-Kit | Multiple | 41.00% | 2024-10-30 | unknown |
| OrcaLoca + Agentless-1.5 + Claude-3.5 Sonnet (20241022) | OrcaLoca + Agentless-1.5 | Claude 3.5 Sonnet | 41.00% | 2025-01-13 | unknown |
| Agentless-1.5 + Claude-3.5 Sonnet (20241022) | Agentless-1.5 | Claude 3.5 Sonnet | 40.67% | 2024-12-02 | unknown |
| OpenCSG Starship Agentic Coder + GPT 4 (0806) | OpenCSG Starship Agentic Coder | GPT-4 (0806) | 39.67% | 2025-01-13 | unknown |
| Bytedance MarsCode Agent | Bytedance MarsCode Agent | Undisclosed | 39.33% | 2024-09-12 | unknown |
| Moatless Tools + Claude 3.5 Sonnet (20241022) | Moatless Tools | Claude 3.5 Sonnet | 39.00% | 2025-01-14 | unknown |
| Honeycomb | Honeycomb | Undisclosed | 38.33% | 2024-08-20 | unknown |
| Moatless Tools + Claude 3.5 Sonnet (20241022) | Moatless Tools | Claude 3.5 Sonnet | 38.33% | 2024-11-17 | unknown |
| AbanteAI MentatBot + GPT 4o (2024-05-13) | AbanteAI MentatBot | GPT-4o | 38.00% | 2024-06-27 | unknown |
| Patched.Codes Patchwork | Patched.Codes Patchwork | Claude 3.5 Sonnet | 37.00% | 2025-01-04 | unknown |
| KGCompass + DeepSeek V3 | KGCompass | DeepSeek V3 | 36.67% | 2025-06-09 | 2+ |
| AppMap Navie v2 | AppMap Navie v2 | Multiple | 36.00% | 2024-11-13 | unknown |
| Gru(2024-08-11) | Gru | Undisclosed | 35.67% | 2024-08-11 | unknown |
| CodeFuse-AAIS | CodeFuse-AAIS | Undisclosed | 35.67% | 2025-01-04 | unknown |
| Isoform | Isoform | Undisclosed | 35.00% | 2024-08-29 | unknown |
| Bytedance MarsCode Agent + GPT 4o (2024-05-13) | Bytedance MarsCode Agent | GPT-4o | 34.00% | 2024-07-23 | unknown |
| SuperCoder2.0 | SuperCoder2.0 | Undisclosed | 34.00% | 2024-08-06 | unknown |
| Alibaba Lingma Agent | Alibaba Lingma Agent | Undisclosed | 33.00% | 2024-06-22 | unknown |
| Agentless Lite + O3 Mini (20250214) | Agentless Lite | O3 Mini | 32.33% | 2025-02-14 | unknown |
| Agentless-1.5 + GPT 4o (2024-05-13) | Agentless-1.5 | GPT-4o | 32.00% | 2024-10-28 | unknown |
| Factory Code Droid | Factory Code Droid | Undisclosed | 31.33% | 2024-06-17 | unknown |
| CodeShellTester + GPT 4o (2024-05-13) | CodeShellTester | GPT-4o | 31.33% | 2024-11-11 | unknown |
| AutoCodeRover (v20240620) + GPT 4o (2024-05-13) | AutoCodeRover | GPT-4o | 30.67% | 2024-06-21 | unknown |
| Moatless Tools + Deepseek V3 | Moatless Tools | Deepseek V3 | 30.67% | 2025-01-11 | unknown |
| Aegis - o3-mini_1.0 | Aegis | o3-mini_1.0 | 30.33% | 2025-02-07 | unknown |
| AIGCode Infant-Coder(2024-08-30) | AIGCode Infant-Coder | Undisclosed | 30.00% | 2024-09-08 | unknown |
| Kortix AI (claude-3-5-sonnet-20241022) | Kortix AI | claude-3-5-sonnet-20241022 | 30.00% | 2024-12-03 | unknown |
| Amazon Q Developer Agent (v20240719-dev) | Amazon Q Developer Agent | Undisclosed | 29.67% | 2024-07-21 | unknown |
| Agentless + RepoGraph + GPT-4o | Agentless + RepoGraph | GPT-4o | 29.67% | 2024-08-08 | unknown |
| CodeR + GPT 4 (1106) | CodeR | GPT-4 (1106) | 28.33% | 2024-06-04 | unknown |
| reproducedRG | reproducedRG | GPT-4o | 28.00% | 2024-11-17 | unknown |
| SIMA + GPT 4o (2024-05-13) | SIMA | GPT-4o | 27.67% | 2024-07-06 | unknown |
| MASAI + GPT 4o (2024-05-13) | MASAI | GPT-4o | 27.33% | 2024-06-12 | unknown |
| Agentless + GPT 4o (2024-05-13) | Agentless | GPT-4o | 27.33% | 2024-06-30 | unknown |
| IBM Research Agent-101 | IBM Research Agent-101 | Undisclosed | 26.67% | 2024-06-12 | unknown |
| Moatless Tools + Claude 3.5 Sonnet | Moatless Tools | Claude 3.5 Sonnet | 26.67% | 2024-06-23 | unknown |
| OpenHands + CodeAct v1.8 | OpenHands | CodeAct v1.8 | 26.67% | 2024-07-25 | unknown |
| Aider + GPT 4o & Claude 3 Opus | Aider | GPT-4o & Claude 3 Opus | 26.33% | 2024-05-23 | unknown |
| HyperAgent | HyperAgent | GPT-4o | 25.33% | 2024-09-25 | unknown |
| Moatless Tools + GPT 4o (2024-05-13) | Moatless Tools | GPT-4o | 24.67% | 2024-06-17 | unknown |
| SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) | SWE-Fixer | Qwen2.5 (7B + 72B) | 24.67% | 2025-03-06 | unknown |
| OpenCSG StarShip CodeGenAgent + GPT 4 (0613) | OpenCSG StarShip CodeGenAgent | GPT-4 (0613) | 23.67% | 2024-05-24 | unknown |
| IBM AI Agent SWE-1.0 (with open LLMs) | IBM AI Agent SWE-1.0 | Multiple | 23.67% | 2024-10-16 | unknown |
| SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) 20241128 | SWE-Fixer | Qwen2.5 (7B + 72B) | 23.33% | 2024-11-28 | unknown |
| SWE-agent + Claude 3.5 Sonnet | SWE-agent | Claude 3.5 Sonnet | 23.00% | 2024-06-20 | unknown |
| AppMap Navie + GPT 4o (2024-05-13) | AppMap Navie | GPT-4o | 21.67% | 2024-06-15 | unknown |
| Bytedance AutoSE (based on SWE-Agent) + GPT4/GPT4o Mixed (20240828) | Bytedance AutoSE | Multiple | 21.67% | 2024-08-28 | unknown |
| Amazon Q Developer Agent (v20240430-dev) | Amazon Q Developer Agent | Undisclosed | 20.33% | 2024-05-09 | unknown |
| AutoCodeRover (v20240408) + GPT 4 (0125) | AutoCodeRover | GPT-4 (0125) | 19.00% | 2024-05-30 | unknown |
| SWE-agent + GPT 4o (2024-05-13) | SWE-agent | GPT-4o | 18.33% | 2024-07-28 | unknown |
| SWE-agent + GPT 4 (1106) | SWE-agent | GPT-4 (1106) | 18.00% | 2024-04-02 | unknown |
| MCTS-Refine-7B | MCTS-Refine-7B | MCTS Refine 7B | 16.33% | 2025-06-27 | 1 |
| SWE-agent + Claude 3 Opus | SWE-agent | Claude 3 Opus | 11.67% | 2024-04-02 | unknown |
| RAG + Claude 3 Opus | RAG baseline | Claude 3 Opus | 4.33% | 2024-04-02 | unknown |
| RAG + Claude 2 | RAG baseline | Claude 2 | 3.00% | 2023-10-10 | unknown |
| RAG + GPT 4 (1106) | RAG baseline | GPT-4 (1106) | 2.67% | 2024-04-02 | unknown |
| RAG + SWE-Llama 7B | RAG baseline | SWE-Llama 7B | 1.33% | 2023-10-10 | unknown |
| RAG + SWE-Llama 13B | RAG baseline | SWE-Llama 13B | 1.00% | 2023-10-10 | unknown |
| RAG + ChatGPT 3.5 | RAG baseline | GPT-3.5 | 0.33% | 2023-10-10 | unknown |
| 0 of 84 plotted entries score higher than our highest result; 14 of 84 score lower than our lowest result. | |||||
Official leaderboard
The saved official snapshot contains no exact model-name matches for our 12 included models. We do not substitute base models for modified or quantized routes.
Official entries can use different agents, budgets, tools and test subsets. These results are not an official leaderboard submission or a controlled comparison against those entries. Snapshot retrieved 2026-10-29T17:45:14.442143+00:00 at commit 193160a463a435d05cf44a1fa9dc5eac832113c3.
Official SWE-bench leaderboard · Saved comparison · Read the protocol