Spec KittywithMirantis TokenFactoryandexe.dev
Benchmarks

Mirantis TokenFactory · SWE-bench

kimi-k3

49.30% Final

1,131 tasks solved inside our limits. 2,294 of 2,294 Full attempts graded (100.00%).

49.30%47.26%–51.35%
Full · solved inside our limits
ModelSolvedCut off by limitsContext capTime limit
kimi-k349.30%22.80% (523 / 2,294) · limit-constrained6.41%16.39%

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Outcome mix

Report statuses partition the Full set. Empty patches and reviewed timeouts overlap unresolved outcomes; they are not extra attempts. Incident reruns are not an outcome category.

Full outcomes and overlapping evidence annotations
OutcomeCount
Resolved1,131
Unresolved1,163
Ungraded0
Missing0
Empty patches (grade reason)525
Adjudicated outer timeouts28
Reviewed test timeouts0
Outage rerunsUnavailable: no active-attempt rerun manifest in the supplied inputs

By repository

Full tasks by repository
RepositoryResolved / gradedResolvedCut off by limitsCoverage
astropy/astropy24 / 9525.26%52.63% (50 / 95) · limit-constrained100.00%
django/django506 / 85059.53%20.82% (177 / 850) · limit-constrained100.00%
matplotlib/matplotlib97 / 18452.72%14.67% (27 / 184)100.00%
mwaskom/seaborn10 / 2245.45%18.18% (4 / 22)100.00%
pallets/flask8 / 1172.73%9.09% (1 / 11)100.00%
psf/requests12 / 4427.27%6.82% (3 / 44)100.00%
pydata/xarray46 / 11041.82%19.09% (21 / 110)100.00%
pylint-dev/pylint19 / 5733.33%24.56% (14 / 57) · limit-constrained100.00%
pytest-dev/pytest64 / 11953.78%10.92% (13 / 119)100.00%
scikit-learn/scikit-learn125 / 22954.59%12.23% (28 / 229)100.00%
sphinx-doc/sphinx85 / 18745.45%26.20% (49 / 187) · limit-constrained100.00%
sympy/sympy135 / 38634.97%35.23% (136 / 386) · limit-constrained100.00%

Tokens and steps

Read 2,294 trajectories from 2,294 collected attempt directories. Steps use info.model_stats.api_calls. Tokens sum recorded assistant response usage per attempt; prompt tokens include repeated context. Missing fields remain unknown. Partial attempts and partial usage are included as observed totals.

180 inspected trajectories lack prompt or completion usage for some recorded calls. These totals are lower bounds. Download usage summaries and per-attempt metrics.

Steps

Median 34 · 2,294 observed

5 to 17.4: 179 attempts17.4 to 29.8: 725 attempts29.8 to 42.2: 639 attempts42.2 to 54.6: 344 attempts54.6 to 67: 191 attempts67 to 79.4: 138 attempts79.4 to 91.8: 51 attempts91.8 to 104.2: 20 attempts104.2 to 116.6: 4 attempts116.6 to 129: 3 attempts5129
Read bin counts

5–17.4: 179; 17.4–29.8: 725; 29.8–42.2: 639; 42.2–54.6: 344; 54.6–67: 191; 67–79.4: 138; 79.4–91.8: 51; 91.8–104.2: 20; 104.2–116.6: 4; 116.6–129: 3

Prompt tokens

Median 822,949 · 2,294 observed

11,918 to 820,960.9: 1,144 attempts820,960.9 to 1,630,003.8: 506 attempts1,630,003.8 to 2,439,046.7: 263 attempts2,439,046.7 to 3,248,089.6: 165 attempts3,248,089.6 to 4,057,132.5: 86 attempts4,057,132.5 to 4,866,175.4: 58 attempts4,866,175.4 to 5,675,218.3: 46 attempts5,675,218.3 to 6,484,261.2: 16 attempts6,484,261.2 to 7,293,304.1: 5 attempts7,293,304.1 to 8,102,347: 5 attempts11,9188,102,347
Read bin counts

11,918–820,960.9: 1,144; 820,960.9–1,630,003.8: 506; 1,630,003.8–2,439,046.7: 263; 2,439,046.7–3,248,089.6: 165; 3,248,089.6–4,057,132.5: 86; 4,057,132.5–4,866,175.4: 58; 4,866,175.4–5,675,218.3: 46; 5,675,218.3–6,484,261.2: 16; 6,484,261.2–7,293,304.1: 5; 7,293,304.1–8,102,347: 5

Completion tokens

Median 25,917.5 · 2,294 observed

1,041 to 12,918.5: 667 attempts12,918.5 to 24,796: 452 attempts24,796 to 36,673.5: 357 attempts36,673.5 to 48,551: 249 attempts48,551 to 60,428.5: 180 attempts60,428.5 to 72,306: 140 attempts72,306 to 84,183.5: 106 attempts84,183.5 to 96,061: 94 attempts96,061 to 107,938.5: 38 attempts107,938.5 to 119,816: 11 attempts1,041119,816
Read bin counts

1,041–12,918.5: 667; 12,918.5–24,796: 452; 24,796–36,673.5: 357; 36,673.5–48,551: 249; 48,551–60,428.5: 180; 60,428.5–72,306: 140; 72,306–84,183.5: 106; 84,183.5–96,061: 94; 96,061–107,938.5: 38; 107,938.5–119,816: 11

Total tokens

Median 850,302 · 2,294 observed

12,959 to 828,934.9: 1,127 attempts828,934.9 to 1,644,910.8: 509 attempts1,644,910.8 to 2,460,886.7: 270 attempts2,460,886.7 to 3,276,862.6: 164 attempts3,276,862.6 to 4,092,838.5: 89 attempts4,092,838.5 to 4,908,814.4: 63 attempts4,908,814.4 to 5,724,790.3: 46 attempts5,724,790.3 to 6,540,766.2: 16 attempts6,540,766.2 to 7,356,742.1: 5 attempts7,356,742.1 to 8,172,718: 5 attempts12,9598,172,718
Read bin counts

12,959–828,934.9: 1,127; 828,934.9–1,644,910.8: 509; 1,644,910.8–2,460,886.7: 270; 2,460,886.7–3,276,862.6: 164; 3,276,862.6–4,092,838.5: 89; 4,092,838.5–4,908,814.4: 63; 4,908,814.4–5,724,790.3: 46; 5,724,790.3–6,540,766.2: 16; 6,540,766.2–7,356,742.1: 5; 7,356,742.1–8,172,718: 5

Notable events

Gateway-stall timeout attribution is unavailable in the supplied report. We observed 28 outer-process timeout exits and 348 wall-budget exits; 28 outer timeouts have explicit adjudications. These counts do not prove that every timeout was caused by the gateway.

Recorded generation exits
Exit statusCount
CompletionBudgetExceeded1
ContextBudgetExceeded147
OuterProcessTimeout28
RepeatedFormatError1
Submitted1,769
TimeExceeded348

Attempt evidence

Files become downloadable when the evidence packer has produced them. Paths follow evidence/models/<model>/attempts/<repo-slug>/<instance_id>.json.gz. Repository slugs use the SWE-bench double-underscore form. Evidence and verification.

astropy/astropy · 95 tasks
django/django · 850 tasks
matplotlib/matplotlib · 184 tasks
mwaskom/seaborn · 22 tasks
pallets/flask · 11 tasks
psf/requests · 44 tasks
pydata/xarray · 110 tasks
pylint-dev/pylint · 57 tasks
pytest-dev/pytest · 119 tasks
scikit-learn/scikit-learn · 229 tasks
sphinx-doc/sphinx · 187 tasks
sympy/sympy · 386 tasks