Spec KittywithMirantis TokenFactoryandexe.dev
Benchmarks

Mirantis TokenFactory · SWE-bench

nemotron-3.5-lightning

17.00% Final

390 tasks solved inside our limits. 2,294 of 2,294 Full attempts graded (100.00%).

17.00%15.52%–18.59%
Full · solved inside our limits
ModelSolvedCut off by limitsContext capTime limit
nemotron-3.5-lightning17.00%6.41% (147 / 2,294)6.36%0.04%

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Outcome mix

Report statuses partition the Full set. Empty patches and reviewed timeouts overlap unresolved outcomes; they are not extra attempts. Incident reruns are not an outcome category.

Full outcomes and overlapping evidence annotations
OutcomeCount
Resolved390
Unresolved1,904
Ungraded0
Missing0
Empty patches (grade reason)193
Adjudicated outer timeouts0
Reviewed test timeouts0
Outage rerunsUnavailable: no active-attempt rerun manifest in the supplied inputs

By repository

Full tasks by repository
RepositoryResolved / gradedResolvedCut off by limitsCoverage
astropy/astropy15 / 9515.79%3.16% (3 / 95)100.00%
django/django129 / 85015.18%4.47% (38 / 850)100.00%
matplotlib/matplotlib43 / 18423.37%7.07% (13 / 184)100.00%
mwaskom/seaborn5 / 2222.73%9.09% (2 / 22)100.00%
pallets/flask3 / 1127.27%9.09% (1 / 11)100.00%
psf/requests5 / 4411.36%4.55% (2 / 44)100.00%
pydata/xarray19 / 11017.27%8.18% (9 / 110)100.00%
pylint-dev/pylint4 / 577.02%8.77% (5 / 57)100.00%
pytest-dev/pytest28 / 11923.53%7.56% (9 / 119)100.00%
scikit-learn/scikit-learn42 / 22918.34%3.06% (7 / 229)100.00%
sphinx-doc/sphinx25 / 18713.37%17.11% (32 / 187)100.00%
sympy/sympy72 / 38618.65%6.74% (26 / 386)100.00%

Tokens and steps

Read 2,294 trajectories from 2,294 collected attempt directories. Steps use info.model_stats.api_calls. Tokens sum recorded assistant response usage per attempt; prompt tokens include repeated context. Missing fields remain unknown. Partial attempts and partial usage are included as observed totals.

232 inspected trajectories lack prompt or completion usage for some recorded calls. These totals are lower bounds. Download usage summaries and per-attempt metrics.

Steps

Median 74 · 2,294 observed

9 to 33.1: 288 attempts33.1 to 57.2: 515 attempts57.2 to 81.3: 458 attempts81.3 to 105.4: 357 attempts105.4 to 129.5: 248 attempts129.5 to 153.6: 172 attempts153.6 to 177.7: 130 attempts177.7 to 201.8: 83 attempts201.8 to 225.9: 29 attempts225.9 to 250: 14 attempts9250
Read bin counts

9–33.1: 288; 33.1–57.2: 515; 57.2–81.3: 458; 81.3–105.4: 357; 105.4–129.5: 248; 129.5–153.6: 172; 153.6–177.7: 130; 177.7–201.8: 83; 201.8–225.9: 29; 225.9–250: 14

Prompt tokens

Median 1,678,275 · 2,294 observed

24,246 to 1,339,853.7: 992 attempts1,339,853.7 to 2,655,461.4: 476 attempts2,655,461.4 to 3,971,069.1: 274 attempts3,971,069.1 to 5,286,676.8: 166 attempts5,286,676.8 to 6,602,284.5: 106 attempts6,602,284.5 to 7,917,892.2: 101 attempts7,917,892.2 to 9,233,499.9: 87 attempts9,233,499.9 to 10,549,107.6: 52 attempts10,549,107.6 to 11,864,715.3: 26 attempts11,864,715.3 to 13,180,323: 14 attempts24,24613,180,323
Read bin counts

24,246–1,339,853.7: 992; 1,339,853.7–2,655,461.4: 476; 2,655,461.4–3,971,069.1: 274; 3,971,069.1–5,286,676.8: 166; 5,286,676.8–6,602,284.5: 106; 6,602,284.5–7,917,892.2: 101; 7,917,892.2–9,233,499.9: 87; 9,233,499.9–10,549,107.6: 52; 10,549,107.6–11,864,715.3: 26; 11,864,715.3–13,180,323: 14

Completion tokens

Median 20,213.5 · 2,294 observed

817 to 10,720.1: 542 attempts10,720.1 to 20,623.2: 630 attempts20,623.2 to 30,526.3: 435 attempts30,526.3 to 40,429.4: 252 attempts40,429.4 to 50,332.5: 179 attempts50,332.5 to 60,235.6: 110 attempts60,235.6 to 70,138.7: 85 attempts70,138.7 to 80,041.8: 44 attempts80,041.8 to 89,944.9: 11 attempts89,944.9 to 99,848: 6 attempts81799,848
Read bin counts

817–10,720.1: 542; 10,720.1–20,623.2: 630; 20,623.2–30,526.3: 435; 30,526.3–40,429.4: 252; 40,429.4–50,332.5: 179; 50,332.5–60,235.6: 110; 60,235.6–70,138.7: 85; 70,138.7–80,041.8: 44; 80,041.8–89,944.9: 11; 89,944.9–99,848: 6

Total tokens

Median 1,702,121 · 2,294 observed

25,063 to 1,344,629: 985 attempts1,344,629 to 2,664,195: 476 attempts2,664,195 to 3,983,761: 279 attempts3,983,761 to 5,303,327: 165 attempts5,303,327 to 6,622,893: 107 attempts6,622,893 to 7,942,459: 102 attempts7,942,459 to 9,262,025: 86 attempts9,262,025 to 10,581,591: 54 attempts10,581,591 to 11,901,157: 26 attempts11,901,157 to 13,220,723: 14 attempts25,06313,220,723
Read bin counts

25,063–1,344,629: 985; 1,344,629–2,664,195: 476; 2,664,195–3,983,761: 279; 3,983,761–5,303,327: 165; 5,303,327–6,622,893: 107; 6,622,893–7,942,459: 102; 7,942,459–9,262,025: 86; 9,262,025–10,581,591: 54; 10,581,591–11,901,157: 26; 11,901,157–13,220,723: 14

Notable events

Task-wide grading amendments and reviewed timeout policies apply. See the methodology and each attempt’s evidence.

Recorded generation exits
Exit statusCount
ContextBudgetExceeded146
LimitsExceeded5
Submitted2,142
TimeExceeded1

Attempt evidence

Files become downloadable when the evidence packer has produced them. Paths follow evidence/models/<model>/attempts/<repo-slug>/<instance_id>.json.gz. Repository slugs use the SWE-bench double-underscore form. Evidence and verification.

astropy/astropy · 95 tasks
django/django · 850 tasks
matplotlib/matplotlib · 184 tasks
mwaskom/seaborn · 22 tasks
pallets/flask · 11 tasks
psf/requests · 44 tasks
pydata/xarray · 110 tasks
pylint-dev/pylint · 57 tasks
pytest-dev/pytest · 119 tasks
scikit-learn/scikit-learn · 229 tasks
sphinx-doc/sphinx · 187 tasks
sympy/sympy · 386 tasks