Spec KittywithMirantis TokenFactoryandexe.dev
Benchmarks

Mirantis TokenFactory · SWE-bench

Compare paired results

Scores show what each model solved inside our limits.

Full · solved inside our limits
ModelSolvedCut off by limitsContext capTime limit
deepseek-v4.1-flash-101754.97%5.01% (115 / 2,294)5.01%0.00%
deepseek-v4.1-flash-1017-uncensored-fp852.62%5.19% (119 / 2,294)4.36%0.83%
glm-5.352.01%5.32% (122 / 2,294)4.71%0.61%
glm-5.3-flash48.69%11.60% (266 / 2,294)0.35%11.25%
glm-5.3-flash-uncensored-fp850.09%8.15% (187 / 2,294)7.85%0.31%
glm-5.3-uncensored-fp834.09%45.95% (1,054 / 2,294) · limit-constrained45.64%0.31%
kimi-k349.30%22.80% (523 / 2,294) · limit-constrained6.41%16.39%
kimi-k3-abliterated38.32%45.03% (1,033 / 2,294) · limit-constrained1.05%43.98%
minimax-m3.1-flash16.39%75.50% (1,732 / 2,294) · limit-constrained75.46%0.04%
nemotron-3-super31.52%6.36% (146 / 2,294)6.36%0.00%
nemotron-3-ultra40.76%4.36% (100 / 2,294)4.23%0.13%
nemotron-3.5-lightning17.00%6.41% (147 / 2,294)6.36%0.04%

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Compare which tasks models solved inside our limits, and where their results differ.

Resolved by repository

Cells show resolved / graded as a percentage. Each cell includes its graded denominator. Pending attempts are excluded. Ink density follows the percentage; text carries the value.

astropy/astropydjango/djangomatplotlib/matplotlibmwaskom/seabornpallets/flaskpsf/requestspydata/xarraypylint-dev/pylintpytest-dev/pytestscikit-learn/scikit-learnsphinx-doc/sphinxsympy/sympydeepseek-v4.1-flash-1017deepseek-v4.1-flash-1017 · astropy/astropy: 47.37%; 95/95 graded; cut off by limits: 8.42% (8 / 95)47.37%deepseek-v4.1-flash-1017 · django/django: 62.82%; 850/850 graded; cut off by limits: 3.88% (33 / 850)62.82%deepseek-v4.1-flash-1017 · matplotlib/matplotlib: 68.48%; 184/184 graded; cut off by limits: 2.17% (4 / 184)68.48%deepseek-v4.1-flash-1017 · mwaskom/seaborn: 59.09%; 22/22 graded; cut off by limits: 0.00% (0 / 22)59.09%deepseek-v4.1-flash-1017 · pallets/flask: 72.73%; 11/11 graded; cut off by limits: 0.00% (0 / 11)72.73%deepseek-v4.1-flash-1017 · psf/requests: 27.27%; 44/44 graded; cut off by limits: 0.00% (0 / 44)27.27%deepseek-v4.1-flash-1017 · pydata/xarray: 50.91%; 110/110 graded; cut off by limits: 8.18% (9 / 110)50.91%deepseek-v4.1-flash-1017 · pylint-dev/pylint: 35.09%; 57/57 graded; cut off by limits: 12.28% (7 / 57)35.09%deepseek-v4.1-flash-1017 · pytest-dev/pytest: 55.46%; 119/119 graded; cut off by limits: 4.20% (5 / 119)55.46%deepseek-v4.1-flash-1017 · scikit-learn/scikit-learn: 50.22%; 229/229 graded; cut off by limits: 3.49% (8 / 229)50.22%deepseek-v4.1-flash-1017 · sphinx-doc/sphinx: 50.80%; 187/187 graded; cut off by limits: 9.09% (17 / 187)50.80%deepseek-v4.1-flash-1017 · sympy/sympy: 44.30%; 386/386 graded; cut off by limits: 6.22% (24 / 386)44.30%deepseek-v4.1-flash-1017-uncensored-fp8deepseek-v4.1-flash-1017-uncensored-fp8 · astropy/astropy: 44.21%; 95/95 graded; cut off by limits: 12.63% (12 / 95)44.21%deepseek-v4.1-flash-1017-uncensored-fp8 · django/django: 61.29%; 850/850 graded; cut off by limits: 3.41% (29 / 850)61.29%deepseek-v4.1-flash-1017-uncensored-fp8 · matplotlib/matplotlib: 61.41%; 184/184 graded; cut off by limits: 4.35% (8 / 184)61.41%deepseek-v4.1-flash-1017-uncensored-fp8 · mwaskom/seaborn: 59.09%; 22/22 graded; cut off by limits: 0.00% (0 / 22)59.09%deepseek-v4.1-flash-1017-uncensored-fp8 · pallets/flask: 72.73%; 11/11 graded; cut off by limits: 9.09% (1 / 11)72.73%deepseek-v4.1-flash-1017-uncensored-fp8 · psf/requests: 29.55%; 44/44 graded; cut off by limits: 0.00% (0 / 44)29.55%deepseek-v4.1-flash-1017-uncensored-fp8 · pydata/xarray: 44.55%; 110/110 graded; cut off by limits: 7.27% (8 / 110)44.55%deepseek-v4.1-flash-1017-uncensored-fp8 · pylint-dev/pylint: 36.84%; 57/57 graded; cut off by limits: 7.02% (4 / 57)36.84%deepseek-v4.1-flash-1017-uncensored-fp8 · pytest-dev/pytest: 55.46%; 119/119 graded; cut off by limits: 4.20% (5 / 119)55.46%deepseek-v4.1-flash-1017-uncensored-fp8 · scikit-learn/scikit-learn: 48.91%; 229/229 graded; cut off by limits: 0.87% (2 / 229)48.91%deepseek-v4.1-flash-1017-uncensored-fp8 · sphinx-doc/sphinx: 49.20%; 187/187 graded; cut off by limits: 10.16% (19 / 187)49.20%deepseek-v4.1-flash-1017-uncensored-fp8 · sympy/sympy: 40.67%; 386/386 graded; cut off by limits: 8.03% (31 / 386)40.67%glm-5.3glm-5.3 · astropy/astropy: 48.42%; 95/95 graded; cut off by limits: 8.42% (8 / 95)48.42%glm-5.3 · django/django: 57.41%; 850/850 graded; cut off by limits: 5.06% (43 / 850)57.41%glm-5.3 · matplotlib/matplotlib: 60.87%; 184/184 graded; cut off by limits: 4.35% (8 / 184)60.87%glm-5.3 · mwaskom/seaborn: 50.00%; 22/22 graded; cut off by limits: 4.55% (1 / 22)50.00%glm-5.3 · pallets/flask: 54.55%; 11/11 graded; cut off by limits: 0.00% (0 / 11)54.55%glm-5.3 · psf/requests: 27.27%; 44/44 graded; cut off by limits: 0.00% (0 / 44)27.27%glm-5.3 · pydata/xarray: 42.73%; 110/110 graded; cut off by limits: 6.36% (7 / 110)42.73%glm-5.3 · pylint-dev/pylint: 38.60%; 57/57 graded; cut off by limits: 7.02% (4 / 57)38.60%glm-5.3 · pytest-dev/pytest: 55.46%; 119/119 graded; cut off by limits: 4.20% (5 / 119)55.46%glm-5.3 · scikit-learn/scikit-learn: 50.22%; 229/229 graded; cut off by limits: 0.87% (2 / 229)50.22%glm-5.3 · sphinx-doc/sphinx: 50.27%; 187/187 graded; cut off by limits: 10.16% (19 / 187)50.27%glm-5.3 · sympy/sympy: 45.08%; 386/386 graded; cut off by limits: 6.48% (25 / 386)45.08%glm-5.3-flashglm-5.3-flash · astropy/astropy: 45.26%; 95/95 graded; cut off by limits: 5.26% (5 / 95)45.26%glm-5.3-flash · django/django: 53.65%; 850/850 graded; cut off by limits: 14.59% (124 / 850)53.65%glm-5.3-flash · matplotlib/matplotlib: 55.98%; 184/184 graded; cut off by limits: 10.33% (19 / 184)55.98%glm-5.3-flash · mwaskom/seaborn: 45.45%; 22/22 graded; cut off by limits: 18.18% (4 / 22)45.45%glm-5.3-flash · pallets/flask: 54.55%; 11/11 graded; cut off by limits: 0.00% (0 / 11)54.55%glm-5.3-flash · psf/requests: 27.27%; 44/44 graded; cut off by limits: 4.55% (2 / 44)27.27%glm-5.3-flash · pydata/xarray: 39.09%; 110/110 graded; cut off by limits: 15.45% (17 / 110)39.09%glm-5.3-flash · pylint-dev/pylint: 36.84%; 57/57 graded; cut off by limits: 17.54% (10 / 57)36.84%glm-5.3-flash · pytest-dev/pytest: 55.46%; 119/119 graded; cut off by limits: 10.08% (12 / 119)55.46%glm-5.3-flash · scikit-learn/scikit-learn: 44.54%; 229/229 graded; cut off by limits: 6.99% (16 / 229)44.54%glm-5.3-flash · sphinx-doc/sphinx: 47.59%; 187/187 graded; cut off by limits: 13.37% (25 / 187)47.59%glm-5.3-flash · sympy/sympy: 43.01%; 386/386 graded; cut off by limits: 8.29% (32 / 386)43.01%glm-5.3-flash-uncensored-fp8glm-5.3-flash-uncensored-fp8 · astropy/astropy: 42.11%; 95/95 graded; cut off by limits: 13.68% (13 / 95)42.11%glm-5.3-flash-uncensored-fp8 · django/django: 56.71%; 850/850 graded; cut off by limits: 6.59% (56 / 850)56.71%glm-5.3-flash-uncensored-fp8 · matplotlib/matplotlib: 54.89%; 184/184 graded; cut off by limits: 6.52% (12 / 184)54.89%glm-5.3-flash-uncensored-fp8 · mwaskom/seaborn: 45.45%; 22/22 graded; cut off by limits: 13.64% (3 / 22)45.45%glm-5.3-flash-uncensored-fp8 · pallets/flask: 54.55%; 11/11 graded; cut off by limits: 0.00% (0 / 11)54.55%glm-5.3-flash-uncensored-fp8 · psf/requests: 34.09%; 44/44 graded; cut off by limits: 4.55% (2 / 44)34.09%glm-5.3-flash-uncensored-fp8 · pydata/xarray: 41.82%; 110/110 graded; cut off by limits: 10.00% (11 / 110)41.82%glm-5.3-flash-uncensored-fp8 · pylint-dev/pylint: 36.84%; 57/57 graded; cut off by limits: 10.53% (6 / 57)36.84%glm-5.3-flash-uncensored-fp8 · pytest-dev/pytest: 54.62%; 119/119 graded; cut off by limits: 4.20% (5 / 119)54.62%glm-5.3-flash-uncensored-fp8 · scikit-learn/scikit-learn: 48.03%; 229/229 graded; cut off by limits: 4.37% (10 / 229)48.03%glm-5.3-flash-uncensored-fp8 · sphinx-doc/sphinx: 49.73%; 187/187 graded; cut off by limits: 10.16% (19 / 187)49.73%glm-5.3-flash-uncensored-fp8 · sympy/sympy: 41.45%; 386/386 graded; cut off by limits: 12.95% (50 / 386)41.45%glm-5.3-uncensored-fp8glm-5.3-uncensored-fp8 · astropy/astropy: 28.42%; 95/95 graded; cut off by limits: 53.68% (51 / 95) · limit-constrained28.42%glm-5.3-uncensored-fp8 · django/django: 41.65%; 850/850 graded; cut off by limits: 39.88% (339 / 850) · limit-constrained41.65%glm-5.3-uncensored-fp8 · matplotlib/matplotlib: 38.59%; 184/184 graded; cut off by limits: 44.02% (81 / 184) · limit-constrained38.59%glm-5.3-uncensored-fp8 · mwaskom/seaborn: 36.36%; 22/22 graded; cut off by limits: 40.91% (9 / 22) · limit-constrained36.36%glm-5.3-uncensored-fp8 · pallets/flask: 54.55%; 11/11 graded; cut off by limits: 18.18% (2 / 11)54.55%glm-5.3-uncensored-fp8 · psf/requests: 20.45%; 44/44 graded; cut off by limits: 29.55% (13 / 44) · limit-constrained20.45%glm-5.3-uncensored-fp8 · pydata/xarray: 33.64%; 110/110 graded; cut off by limits: 50.91% (56 / 110) · limit-constrained33.64%glm-5.3-uncensored-fp8 · pylint-dev/pylint: 15.79%; 57/57 graded; cut off by limits: 64.91% (37 / 57) · limit-constrained15.79%glm-5.3-uncensored-fp8 · pytest-dev/pytest: 38.66%; 119/119 graded; cut off by limits: 37.82% (45 / 119) · limit-constrained38.66%glm-5.3-uncensored-fp8 · scikit-learn/scikit-learn: 34.50%; 229/229 graded; cut off by limits: 37.55% (86 / 229) · limit-constrained34.50%glm-5.3-uncensored-fp8 · sphinx-doc/sphinx: 25.13%; 187/187 graded; cut off by limits: 59.89% (112 / 187) · limit-constrained25.13%glm-5.3-uncensored-fp8 · sympy/sympy: 23.06%; 386/386 graded; cut off by limits: 57.77% (223 / 386) · limit-constrained23.06%kimi-k3kimi-k3 · astropy/astropy: 25.26%; 95/95 graded; cut off by limits: 52.63% (50 / 95) · limit-constrained25.26%kimi-k3 · django/django: 59.53%; 850/850 graded; cut off by limits: 20.82% (177 / 850) · limit-constrained59.53%kimi-k3 · matplotlib/matplotlib: 52.72%; 184/184 graded; cut off by limits: 14.67% (27 / 184)52.72%kimi-k3 · mwaskom/seaborn: 45.45%; 22/22 graded; cut off by limits: 18.18% (4 / 22)45.45%kimi-k3 · pallets/flask: 72.73%; 11/11 graded; cut off by limits: 9.09% (1 / 11)72.73%kimi-k3 · psf/requests: 27.27%; 44/44 graded; cut off by limits: 6.82% (3 / 44)27.27%kimi-k3 · pydata/xarray: 41.82%; 110/110 graded; cut off by limits: 19.09% (21 / 110)41.82%kimi-k3 · pylint-dev/pylint: 33.33%; 57/57 graded; cut off by limits: 24.56% (14 / 57) · limit-constrained33.33%kimi-k3 · pytest-dev/pytest: 53.78%; 119/119 graded; cut off by limits: 10.92% (13 / 119)53.78%kimi-k3 · scikit-learn/scikit-learn: 54.59%; 229/229 graded; cut off by limits: 12.23% (28 / 229)54.59%kimi-k3 · sphinx-doc/sphinx: 45.45%; 187/187 graded; cut off by limits: 26.20% (49 / 187) · limit-constrained45.45%kimi-k3 · sympy/sympy: 34.97%; 386/386 graded; cut off by limits: 35.23% (136 / 386) · limit-constrained34.97%kimi-k3-abliteratedkimi-k3-abliterated · astropy/astropy: 38.95%; 95/95 graded; cut off by limits: 34.74% (33 / 95) · limit-constrained38.95%kimi-k3-abliterated · django/django: 51.65%; 850/850 graded; cut off by limits: 35.06% (298 / 850) · limit-constrained51.65%kimi-k3-abliterated · matplotlib/matplotlib: 30.43%; 184/184 graded; cut off by limits: 53.26% (98 / 184) · limit-constrained30.43%kimi-k3-abliterated · mwaskom/seaborn: 9.09%; 22/22 graded; cut off by limits: 77.27% (17 / 22) · limit-constrained9.09%kimi-k3-abliterated · pallets/flask: 45.45%; 11/11 graded; cut off by limits: 36.36% (4 / 11) · limit-constrained45.45%kimi-k3-abliterated · psf/requests: 22.73%; 44/44 graded; cut off by limits: 43.18% (19 / 44) · limit-constrained22.73%kimi-k3-abliterated · pydata/xarray: 26.36%; 110/110 graded; cut off by limits: 62.73% (69 / 110) · limit-constrained26.36%kimi-k3-abliterated · pylint-dev/pylint: 12.28%; 57/57 graded; cut off by limits: 77.19% (44 / 57) · limit-constrained12.28%kimi-k3-abliterated · pytest-dev/pytest: 36.13%; 119/119 graded; cut off by limits: 50.42% (60 / 119) · limit-constrained36.13%kimi-k3-abliterated · scikit-learn/scikit-learn: 35.81%; 229/229 graded; cut off by limits: 46.72% (107 / 229) · limit-constrained35.81%kimi-k3-abliterated · sphinx-doc/sphinx: 33.16%; 187/187 graded; cut off by limits: 43.32% (81 / 187) · limit-constrained33.16%kimi-k3-abliterated · sympy/sympy: 27.72%; 386/386 graded; cut off by limits: 52.59% (203 / 386) · limit-constrained27.72%minimax-m3.1-flashminimax-m3.1-flash · astropy/astropy: 22.11%; 95/95 graded; cut off by limits: 69.47% (66 / 95) · limit-constrained22.11%minimax-m3.1-flash · django/django: 20.94%; 850/850 graded; cut off by limits: 69.65% (592 / 850) · limit-constrained20.94%minimax-m3.1-flash · matplotlib/matplotlib: 18.48%; 184/184 graded; cut off by limits: 74.46% (137 / 184) · limit-constrained18.48%minimax-m3.1-flash · mwaskom/seaborn: 18.18%; 22/22 graded; cut off by limits: 72.73% (16 / 22) · limit-constrained18.18%minimax-m3.1-flash · pallets/flask: 36.36%; 11/11 graded; cut off by limits: 45.45% (5 / 11) · limit-constrained36.36%minimax-m3.1-flash · psf/requests: 15.91%; 44/44 graded; cut off by limits: 65.91% (29 / 44) · limit-constrained15.91%minimax-m3.1-flash · pydata/xarray: 18.18%; 110/110 graded; cut off by limits: 77.27% (85 / 110) · limit-constrained18.18%minimax-m3.1-flash · pylint-dev/pylint: 10.53%; 57/57 graded; cut off by limits: 85.96% (49 / 57) · limit-constrained10.53%minimax-m3.1-flash · pytest-dev/pytest: 12.61%; 119/119 graded; cut off by limits: 81.51% (97 / 119) · limit-constrained12.61%minimax-m3.1-flash · scikit-learn/scikit-learn: 13.54%; 229/229 graded; cut off by limits: 75.11% (172 / 229) · limit-constrained13.54%minimax-m3.1-flash · sphinx-doc/sphinx: 5.88%; 187/187 graded; cut off by limits: 92.51% (173 / 187) · limit-constrained5.88%minimax-m3.1-flash · sympy/sympy: 11.66%; 386/386 graded; cut off by limits: 80.57% (311 / 386) · limit-constrained11.66%nemotron-3-supernemotron-3-super · astropy/astropy: 29.47%; 95/95 graded; cut off by limits: 6.32% (6 / 95)29.47%nemotron-3-super · django/django: 39.18%; 850/850 graded; cut off by limits: 4.35% (37 / 850)39.18%nemotron-3-super · matplotlib/matplotlib: 28.26%; 184/184 graded; cut off by limits: 4.89% (9 / 184)28.26%nemotron-3-super · mwaskom/seaborn: 36.36%; 22/22 graded; cut off by limits: 4.55% (1 / 22)36.36%nemotron-3-super · pallets/flask: 18.18%; 11/11 graded; cut off by limits: 0.00% (0 / 11)18.18%nemotron-3-super · psf/requests: 13.64%; 44/44 graded; cut off by limits: 2.27% (1 / 44)13.64%nemotron-3-super · pydata/xarray: 29.09%; 110/110 graded; cut off by limits: 4.55% (5 / 110)29.09%nemotron-3-super · pylint-dev/pylint: 22.81%; 57/57 graded; cut off by limits: 15.79% (9 / 57)22.81%nemotron-3-super · pytest-dev/pytest: 33.61%; 119/119 graded; cut off by limits: 5.88% (7 / 119)33.61%nemotron-3-super · scikit-learn/scikit-learn: 27.95%; 229/229 graded; cut off by limits: 6.55% (15 / 229)27.95%nemotron-3-super · sphinx-doc/sphinx: 23.53%; 187/187 graded; cut off by limits: 12.30% (23 / 187)23.53%nemotron-3-super · sympy/sympy: 26.17%; 386/386 graded; cut off by limits: 8.55% (33 / 386)26.17%nemotron-3-ultranemotron-3-ultra · astropy/astropy: 37.89%; 95/95 graded; cut off by limits: 5.26% (5 / 95)37.89%nemotron-3-ultra · django/django: 47.06%; 850/850 graded; cut off by limits: 4.71% (40 / 850)47.06%nemotron-3-ultra · matplotlib/matplotlib: 44.02%; 184/184 graded; cut off by limits: 2.17% (4 / 184)44.02%nemotron-3-ultra · mwaskom/seaborn: 40.91%; 22/22 graded; cut off by limits: 4.55% (1 / 22)40.91%nemotron-3-ultra · pallets/flask: 45.45%; 11/11 graded; cut off by limits: 0.00% (0 / 11)45.45%nemotron-3-ultra · psf/requests: 18.18%; 44/44 graded; cut off by limits: 0.00% (0 / 44)18.18%nemotron-3-ultra · pydata/xarray: 36.36%; 110/110 graded; cut off by limits: 1.82% (2 / 110)36.36%nemotron-3-ultra · pylint-dev/pylint: 31.58%; 57/57 graded; cut off by limits: 0.00% (0 / 57)31.58%nemotron-3-ultra · pytest-dev/pytest: 45.38%; 119/119 graded; cut off by limits: 5.04% (6 / 119)45.38%nemotron-3-ultra · scikit-learn/scikit-learn: 39.30%; 229/229 graded; cut off by limits: 3.06% (7 / 229)39.30%nemotron-3-ultra · sphinx-doc/sphinx: 31.02%; 187/187 graded; cut off by limits: 7.49% (14 / 187)31.02%nemotron-3-ultra · sympy/sympy: 35.23%; 386/386 graded; cut off by limits: 5.44% (21 / 386)35.23%nemotron-3.5-lightningnemotron-3.5-lightning · astropy/astropy: 15.79%; 95/95 graded; cut off by limits: 3.16% (3 / 95)15.79%nemotron-3.5-lightning · django/django: 15.18%; 850/850 graded; cut off by limits: 4.47% (38 / 850)15.18%nemotron-3.5-lightning · matplotlib/matplotlib: 23.37%; 184/184 graded; cut off by limits: 7.07% (13 / 184)23.37%nemotron-3.5-lightning · mwaskom/seaborn: 22.73%; 22/22 graded; cut off by limits: 9.09% (2 / 22)22.73%nemotron-3.5-lightning · pallets/flask: 27.27%; 11/11 graded; cut off by limits: 9.09% (1 / 11)27.27%nemotron-3.5-lightning · psf/requests: 11.36%; 44/44 graded; cut off by limits: 4.55% (2 / 44)11.36%nemotron-3.5-lightning · pydata/xarray: 17.27%; 110/110 graded; cut off by limits: 8.18% (9 / 110)17.27%nemotron-3.5-lightning · pylint-dev/pylint: 7.02%; 57/57 graded; cut off by limits: 8.77% (5 / 57)7.02%nemotron-3.5-lightning · pytest-dev/pytest: 23.53%; 119/119 graded; cut off by limits: 7.56% (9 / 119)23.53%nemotron-3.5-lightning · scikit-learn/scikit-learn: 18.34%; 229/229 graded; cut off by limits: 3.06% (7 / 229)18.34%nemotron-3.5-lightning · sphinx-doc/sphinx: 13.37%; 187/187 graded; cut off by limits: 17.11% (32 / 187)13.37%nemotron-3.5-lightning · sympy/sympy: 18.65%; 386/386 graded; cut off by limits: 6.74% (26 / 386)18.65%
Read repository values and coverage
Heatmap values: resolved / graded; coverage
Modelastropy/astropyCut off by limitsdjango/djangoCut off by limitsmatplotlib/matplotlibCut off by limitsmwaskom/seabornCut off by limitspallets/flaskCut off by limitspsf/requestsCut off by limitspydata/xarrayCut off by limitspylint-dev/pylintCut off by limitspytest-dev/pytestCut off by limitsscikit-learn/scikit-learnCut off by limitssphinx-doc/sphinxCut off by limitssympy/sympyCut off by limits
deepseek-v4.1-flash-101747.37%45/95; 100.00% coverage8.42% (8 / 95)62.82%534/850; 100.00% coverage3.88% (33 / 850)68.48%126/184; 100.00% coverage2.17% (4 / 184)59.09%13/22; 100.00% coverage0.00% (0 / 22)72.73%8/11; 100.00% coverage0.00% (0 / 11)27.27%12/44; 100.00% coverage0.00% (0 / 44)50.91%56/110; 100.00% coverage8.18% (9 / 110)35.09%20/57; 100.00% coverage12.28% (7 / 57)55.46%66/119; 100.00% coverage4.20% (5 / 119)50.22%115/229; 100.00% coverage3.49% (8 / 229)50.80%95/187; 100.00% coverage9.09% (17 / 187)44.30%171/386; 100.00% coverage6.22% (24 / 386)
deepseek-v4.1-flash-1017-uncensored-fp844.21%42/95; 100.00% coverage12.63% (12 / 95)61.29%521/850; 100.00% coverage3.41% (29 / 850)61.41%113/184; 100.00% coverage4.35% (8 / 184)59.09%13/22; 100.00% coverage0.00% (0 / 22)72.73%8/11; 100.00% coverage9.09% (1 / 11)29.55%13/44; 100.00% coverage0.00% (0 / 44)44.55%49/110; 100.00% coverage7.27% (8 / 110)36.84%21/57; 100.00% coverage7.02% (4 / 57)55.46%66/119; 100.00% coverage4.20% (5 / 119)48.91%112/229; 100.00% coverage0.87% (2 / 229)49.20%92/187; 100.00% coverage10.16% (19 / 187)40.67%157/386; 100.00% coverage8.03% (31 / 386)
glm-5.348.42%46/95; 100.00% coverage8.42% (8 / 95)57.41%488/850; 100.00% coverage5.06% (43 / 850)60.87%112/184; 100.00% coverage4.35% (8 / 184)50.00%11/22; 100.00% coverage4.55% (1 / 22)54.55%6/11; 100.00% coverage0.00% (0 / 11)27.27%12/44; 100.00% coverage0.00% (0 / 44)42.73%47/110; 100.00% coverage6.36% (7 / 110)38.60%22/57; 100.00% coverage7.02% (4 / 57)55.46%66/119; 100.00% coverage4.20% (5 / 119)50.22%115/229; 100.00% coverage0.87% (2 / 229)50.27%94/187; 100.00% coverage10.16% (19 / 187)45.08%174/386; 100.00% coverage6.48% (25 / 386)
glm-5.3-flash45.26%43/95; 100.00% coverage5.26% (5 / 95)53.65%456/850; 100.00% coverage14.59% (124 / 850)55.98%103/184; 100.00% coverage10.33% (19 / 184)45.45%10/22; 100.00% coverage18.18% (4 / 22)54.55%6/11; 100.00% coverage0.00% (0 / 11)27.27%12/44; 100.00% coverage4.55% (2 / 44)39.09%43/110; 100.00% coverage15.45% (17 / 110)36.84%21/57; 100.00% coverage17.54% (10 / 57)55.46%66/119; 100.00% coverage10.08% (12 / 119)44.54%102/229; 100.00% coverage6.99% (16 / 229)47.59%89/187; 100.00% coverage13.37% (25 / 187)43.01%166/386; 100.00% coverage8.29% (32 / 386)
glm-5.3-flash-uncensored-fp842.11%40/95; 100.00% coverage13.68% (13 / 95)56.71%482/850; 100.00% coverage6.59% (56 / 850)54.89%101/184; 100.00% coverage6.52% (12 / 184)45.45%10/22; 100.00% coverage13.64% (3 / 22)54.55%6/11; 100.00% coverage0.00% (0 / 11)34.09%15/44; 100.00% coverage4.55% (2 / 44)41.82%46/110; 100.00% coverage10.00% (11 / 110)36.84%21/57; 100.00% coverage10.53% (6 / 57)54.62%65/119; 100.00% coverage4.20% (5 / 119)48.03%110/229; 100.00% coverage4.37% (10 / 229)49.73%93/187; 100.00% coverage10.16% (19 / 187)41.45%160/386; 100.00% coverage12.95% (50 / 386)
glm-5.3-uncensored-fp828.42%27/95; 100.00% coverage53.68% (51 / 95) · limit-constrained41.65%354/850; 100.00% coverage39.88% (339 / 850) · limit-constrained38.59%71/184; 100.00% coverage44.02% (81 / 184) · limit-constrained36.36%8/22; 100.00% coverage40.91% (9 / 22) · limit-constrained54.55%6/11; 100.00% coverage18.18% (2 / 11)20.45%9/44; 100.00% coverage29.55% (13 / 44) · limit-constrained33.64%37/110; 100.00% coverage50.91% (56 / 110) · limit-constrained15.79%9/57; 100.00% coverage64.91% (37 / 57) · limit-constrained38.66%46/119; 100.00% coverage37.82% (45 / 119) · limit-constrained34.50%79/229; 100.00% coverage37.55% (86 / 229) · limit-constrained25.13%47/187; 100.00% coverage59.89% (112 / 187) · limit-constrained23.06%89/386; 100.00% coverage57.77% (223 / 386) · limit-constrained
kimi-k325.26%24/95; 100.00% coverage52.63% (50 / 95) · limit-constrained59.53%506/850; 100.00% coverage20.82% (177 / 850) · limit-constrained52.72%97/184; 100.00% coverage14.67% (27 / 184)45.45%10/22; 100.00% coverage18.18% (4 / 22)72.73%8/11; 100.00% coverage9.09% (1 / 11)27.27%12/44; 100.00% coverage6.82% (3 / 44)41.82%46/110; 100.00% coverage19.09% (21 / 110)33.33%19/57; 100.00% coverage24.56% (14 / 57) · limit-constrained53.78%64/119; 100.00% coverage10.92% (13 / 119)54.59%125/229; 100.00% coverage12.23% (28 / 229)45.45%85/187; 100.00% coverage26.20% (49 / 187) · limit-constrained34.97%135/386; 100.00% coverage35.23% (136 / 386) · limit-constrained
kimi-k3-abliterated38.95%37/95; 100.00% coverage34.74% (33 / 95) · limit-constrained51.65%439/850; 100.00% coverage35.06% (298 / 850) · limit-constrained30.43%56/184; 100.00% coverage53.26% (98 / 184) · limit-constrained9.09%2/22; 100.00% coverage77.27% (17 / 22) · limit-constrained45.45%5/11; 100.00% coverage36.36% (4 / 11) · limit-constrained22.73%10/44; 100.00% coverage43.18% (19 / 44) · limit-constrained26.36%29/110; 100.00% coverage62.73% (69 / 110) · limit-constrained12.28%7/57; 100.00% coverage77.19% (44 / 57) · limit-constrained36.13%43/119; 100.00% coverage50.42% (60 / 119) · limit-constrained35.81%82/229; 100.00% coverage46.72% (107 / 229) · limit-constrained33.16%62/187; 100.00% coverage43.32% (81 / 187) · limit-constrained27.72%107/386; 100.00% coverage52.59% (203 / 386) · limit-constrained
minimax-m3.1-flash22.11%21/95; 100.00% coverage69.47% (66 / 95) · limit-constrained20.94%178/850; 100.00% coverage69.65% (592 / 850) · limit-constrained18.48%34/184; 100.00% coverage74.46% (137 / 184) · limit-constrained18.18%4/22; 100.00% coverage72.73% (16 / 22) · limit-constrained36.36%4/11; 100.00% coverage45.45% (5 / 11) · limit-constrained15.91%7/44; 100.00% coverage65.91% (29 / 44) · limit-constrained18.18%20/110; 100.00% coverage77.27% (85 / 110) · limit-constrained10.53%6/57; 100.00% coverage85.96% (49 / 57) · limit-constrained12.61%15/119; 100.00% coverage81.51% (97 / 119) · limit-constrained13.54%31/229; 100.00% coverage75.11% (172 / 229) · limit-constrained5.88%11/187; 100.00% coverage92.51% (173 / 187) · limit-constrained11.66%45/386; 100.00% coverage80.57% (311 / 386) · limit-constrained
nemotron-3-super29.47%28/95; 100.00% coverage6.32% (6 / 95)39.18%333/850; 100.00% coverage4.35% (37 / 850)28.26%52/184; 100.00% coverage4.89% (9 / 184)36.36%8/22; 100.00% coverage4.55% (1 / 22)18.18%2/11; 100.00% coverage0.00% (0 / 11)13.64%6/44; 100.00% coverage2.27% (1 / 44)29.09%32/110; 100.00% coverage4.55% (5 / 110)22.81%13/57; 100.00% coverage15.79% (9 / 57)33.61%40/119; 100.00% coverage5.88% (7 / 119)27.95%64/229; 100.00% coverage6.55% (15 / 229)23.53%44/187; 100.00% coverage12.30% (23 / 187)26.17%101/386; 100.00% coverage8.55% (33 / 386)
nemotron-3-ultra37.89%36/95; 100.00% coverage5.26% (5 / 95)47.06%400/850; 100.00% coverage4.71% (40 / 850)44.02%81/184; 100.00% coverage2.17% (4 / 184)40.91%9/22; 100.00% coverage4.55% (1 / 22)45.45%5/11; 100.00% coverage0.00% (0 / 11)18.18%8/44; 100.00% coverage0.00% (0 / 44)36.36%40/110; 100.00% coverage1.82% (2 / 110)31.58%18/57; 100.00% coverage0.00% (0 / 57)45.38%54/119; 100.00% coverage5.04% (6 / 119)39.30%90/229; 100.00% coverage3.06% (7 / 229)31.02%58/187; 100.00% coverage7.49% (14 / 187)35.23%136/386; 100.00% coverage5.44% (21 / 386)
nemotron-3.5-lightning15.79%15/95; 100.00% coverage3.16% (3 / 95)15.18%129/850; 100.00% coverage4.47% (38 / 850)23.37%43/184; 100.00% coverage7.07% (13 / 184)22.73%5/22; 100.00% coverage9.09% (2 / 22)27.27%3/11; 100.00% coverage9.09% (1 / 11)11.36%5/44; 100.00% coverage4.55% (2 / 44)17.27%19/110; 100.00% coverage8.18% (9 / 110)7.02%4/57; 100.00% coverage8.77% (5 / 57)23.53%28/119; 100.00% coverage7.56% (9 / 119)18.34%42/229; 100.00% coverage3.06% (7 / 229)13.37%25/187; 100.00% coverage17.11% (32 / 187)18.65%72/386; 100.00% coverage6.74% (26 / 386)

Paired differences

Exact two-sided McNemar tests use only tasks with graded attempts from both models. The null gives each discordant direction equal probability. We double the smaller binomial tail and cap it at unity; no continuity correction or multiple-comparison adjustment. These exploratory tests do not estimate repeat-run reliability. Pending attempts are not failures.

McNemar p-values: A = row; B = column
Model A / model Bdeepseek-v4.1-flash-1017deepseek-v4.1-flash-1017-uncensored-fp8glm-5.3glm-5.3-flashglm-5.3-flash-uncensored-fp8glm-5.3-uncensored-fp8kimi-k3kimi-k3-abliteratedminimax-m3.1-flashnemotron-3-supernemotron-3-ultranemotron-3.5-lightning
deepseek-v4.1-flash-1017—0.0005946A only 147 · B only 93
Paired 2294		0.0001631A only 193 · B only 125
Paired 2294		2.58e-14A only 253 · B only 109
Paired 2294		7.773e-10A only 222 · B only 110
Paired 2294		5.652e-102A only 529 · B only 50
Paired 2294		5.344e-10A only 284 · B only 154
Paired 2294		1.352e-58A only 492 · B only 110
Paired 2294		5.105e-231A only 912 · B only 27
Paired 2294		4.671e-122A only 580 · B only 42
Paired 2294		9.991e-59A only 388 · B only 62
Paired 2294		6.79e-233A only 892 · B only 21
Paired 2294
deepseek-v4.1-flash-1017-uncensored-fp80.0005946A only 93 · B only 147
Paired 2294		—0.4702A only 169 · B only 155
Paired 2294		1.505e-06A only 218 · B only 128
Paired 2294		0.001006A only 180 · B only 122
Paired 2294		4.737e-82A only 489 · B only 64
Paired 2294		0.0002043A only 243 · B only 167
Paired 2294		6.026e-44A only 457 · B only 129
Paired 2294		6.756e-212A only 862 · B only 31
Paired 2294		2.67e-104A only 532 · B only 48
Paired 2294		2.447e-44A only 341 · B only 69
Paired 2294		2.314e-212A only 843 · B only 26
Paired 2294
glm-5.30.0001631A only 125 · B only 193
Paired 2294		0.4702A only 155 · B only 169
Paired 2294		—2.857e-05A only 200 · B only 124
Paired 2294		0.01291A only 172 · B only 128
Paired 2294		1.364e-82A only 465 · B only 54
Paired 2294		0.002675A only 238 · B only 176
Paired 2294		2.232e-42A only 436 · B only 122
Paired 2294		2.261e-205A only 851 · B only 34
Paired 2294		4.004e-101A only 517 · B only 47
Paired 2294		3.067e-42A only 323 · B only 65
Paired 2294		3.317e-210A only 827 · B only 24
Paired 2294
glm-5.3-flash2.58e-14A only 109 · B only 253
Paired 2294		1.505e-06A only 128 · B only 218
Paired 2294		2.857e-05A only 124 · B only 200
Paired 2294		—0.05713A only 117 · B only 149
Paired 2294		7.888e-56A only 414 · B only 79
Paired 2294		0.5327A only 210 · B only 224
Paired 2294		6.336e-27A only 372 · B only 134
Paired 2294		7.619e-184A only 775 · B only 34
Paired 2294		1.187e-72A only 463 · B only 69
Paired 2294		5.755e-21A only 284 · B only 102
Paired 2294		4.058e-179A only 762 · B only 35
Paired 2294
glm-5.3-flash-uncensored-fp87.773e-10A only 110 · B only 222
Paired 2294		0.001006A only 122 · B only 180
Paired 2294		0.01291A only 128 · B only 172
Paired 2294		0.05713A only 149 · B only 117
Paired 2294		—1.031e-63A only 444 · B only 77
Paired 2294		0.4068A only 219 · B only 201
Paired 2294		1.014e-31A only 410 · B only 140
Paired 2294		1.595e-194A only 805 · B only 32
Paired 2294		2.098e-86A only 480 · B only 54
Paired 2294		1.087e-29A only 295 · B only 81
Paired 2294		8.408e-195A only 786 · B only 27
Paired 2294
glm-5.3-uncensored-fp85.652e-102A only 50 · B only 529
Paired 2294		4.737e-82A only 64 · B only 489
Paired 2294		1.364e-82A only 54 · B only 465
Paired 2294		7.888e-56A only 79 · B only 414
Paired 2294		1.031e-63A only 77 · B only 444
Paired 2294		—3.49e-58A only 82 · B only 431
Paired 2294		4.455e-06A only 172 · B only 269
Paired 2294		7.417e-75A only 477 · B only 71
Paired 2294		0.006366A only 256 · B only 197
Paired 2294		9.012e-13A only 154 · B only 307
Paired 2294		5.668e-59A only 510 · B only 118
Paired 2294
kimi-k35.344e-10A only 154 · B only 284
Paired 2294		0.0002043A only 167 · B only 243
Paired 2294		0.002675A only 176 · B only 238
Paired 2294		0.5327A only 224 · B only 210
Paired 2294		0.4068A only 201 · B only 219
Paired 2294		3.49e-58A only 431 · B only 82
Paired 2294		—4.09e-34A only 351 · B only 99
Paired 2294		4.672e-181A only 798 · B only 43
Paired 2294		7.859e-69A only 500 · B only 92
Paired 2294		2.482e-19A only 340 · B only 144
Paired 2294		9.87e-166A only 802 · B only 61
Paired 2294
kimi-k3-abliterated1.352e-58A only 110 · B only 492
Paired 2294		6.026e-44A only 129 · B only 457
Paired 2294		2.232e-42A only 122 · B only 436
Paired 2294		6.336e-27A only 134 · B only 372
Paired 2294		1.014e-31A only 140 · B only 410
Paired 2294		4.455e-06A only 269 · B only 172
Paired 2294		4.09e-34A only 99 · B only 351
Paired 2294		—5.522e-96A only 581 · B only 78
Paired 2294		4.612e-12A only 333 · B only 177
Paired 2294		0.01383A only 222 · B only 278
Paired 2294		2.068e-83A only 595 · B only 106
Paired 2294
minimax-m3.1-flash5.105e-231A only 27 · B only 912
Paired 2294		6.756e-212A only 31 · B only 862
Paired 2294		2.261e-205A only 34 · B only 851
Paired 2294		7.619e-184A only 34 · B only 775
Paired 2294		1.595e-194A only 32 · B only 805
Paired 2294		7.417e-75A only 71 · B only 477
Paired 2294		4.672e-181A only 43 · B only 798
Paired 2294		5.522e-96A only 78 · B only 581
Paired 2294		—5.145e-51A only 111 · B only 458
Paired 2294		1.178e-114A only 67 · B only 626
Paired 2294		0.5587A only 240 · B only 254
Paired 2294
nemotron-3-super4.671e-122A only 42 · B only 580
Paired 2294		2.67e-104A only 48 · B only 532
Paired 2294		4.004e-101A only 47 · B only 517
Paired 2294		1.187e-72A only 69 · B only 463
Paired 2294		2.098e-86A only 54 · B only 480
Paired 2294		0.006366A only 197 · B only 256
Paired 2294		7.859e-69A only 92 · B only 500
Paired 2294		4.612e-12A only 177 · B only 333
Paired 2294		5.145e-51A only 458 · B only 111
Paired 2294		—5.745e-29A only 83 · B only 295
Paired 2294		1.385e-53A only 419 · B only 86
Paired 2294
nemotron-3-ultra9.991e-59A only 62 · B only 388
Paired 2294		2.447e-44A only 69 · B only 341
Paired 2294		3.067e-42A only 65 · B only 323
Paired 2294		5.755e-21A only 102 · B only 284
Paired 2294		1.087e-29A only 81 · B only 295
Paired 2294		9.012e-13A only 307 · B only 154
Paired 2294		2.482e-19A only 144 · B only 340
Paired 2294		0.01383A only 278 · B only 222
Paired 2294		1.178e-114A only 626 · B only 67
Paired 2294		5.745e-29A only 295 · B only 83
Paired 2294		—9.023e-123A only 589 · B only 44
Paired 2294
nemotron-3.5-lightning6.79e-233A only 21 · B only 892
Paired 2294		2.314e-212A only 26 · B only 843
Paired 2294		3.317e-210A only 24 · B only 827
Paired 2294		4.058e-179A only 35 · B only 762
Paired 2294		8.408e-195A only 27 · B only 786
Paired 2294		5.668e-59A only 118 · B only 510
Paired 2294		9.87e-166A only 61 · B only 802
Paired 2294		2.068e-83A only 106 · B only 595
Paired 2294		0.5587A only 254 · B only 240
Paired 2294		1.385e-53A only 86 · B only 419
Paired 2294		9.023e-123A only 44 · B only 589
Paired 2294		—

Download paired counts and p-values