Mirantis TokenFactory · SWE-bench

glm-5.3-uncensored-fp8

34.09% Final

782 tasks solved inside our limits. 2,294 of 2,294 Full attempts graded (100.00%).

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Outcome mix

Report statuses partition the Full set. Empty patches and reviewed timeouts overlap unresolved outcomes; they are not extra attempts. Incident reruns are not an outcome category.

By repository

Tokens and steps

Read 2,294 trajectories from 2,294 collected attempt directories. Steps use info.model_stats.api_calls. Tokens sum recorded assistant response usage per attempt; prompt tokens include repeated context. Missing fields remain unknown. Partial attempts and partial usage are included as observed totals.

1,318 inspected trajectories lack prompt or completion usage for some recorded calls. These totals are lower bounds. Download usage summaries and per-attempt metrics.

Notable events

The experiment context reports a backend outage followed by fresh reruns. Active-attempt exports do not identify the affected rerun population, so its count is unavailable. Superseded attempts are excluded.

Attempt evidence

Files become downloadable when the evidence packer has produced them. Paths follow evidence/models/<model>/attempts/<repo-slug>/<instance_id>.json.gz. Repository slugs use the SWE-bench double-underscore form. Evidence and verification.