Mirantis TokenFactory · SWE-bench

glm-5.3-flash

48.69% Final

1,117 tasks solved inside our limits. 2,294 of 2,294 Full attempts graded (100.00%).

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Outcome mix

Report statuses partition the Full set. Empty patches and reviewed timeouts overlap unresolved outcomes; they are not extra attempts. Incident reruns are not an outcome category.

By repository

Tokens and steps

Read 2,294 trajectories from 2,294 collected attempt directories. Steps use info.model_stats.api_calls. Tokens sum recorded assistant response usage per attempt; prompt tokens include repeated context. Missing fields remain unknown. Partial attempts and partial usage are included as observed totals.

24 inspected trajectories lack prompt or completion usage for some recorded calls. These totals are lower bounds. Download usage summaries and per-attempt metrics.

Notable events

Restart with a per-request nonce. Gateway route returned corrupted or non-terminating output for repeated identical prefixes; 0/72 failures with a per-request nonce versus about one third without. See STATUS.md. all attempts after the 2026-10-31 restart; earlier attempts are archived as superseded The current scores exclude archived pre-restart attempts.

Gateway route corrupted output for repeated identical prefixes. Restarted 2026-10-31 with a unique per-request leading token (see configs/request-nonce.json); pilot on one worker first, then the fleet. Pending attempts for tasks the other models had already finished (1,517 jobs) are temporarily deferred (status paused) so the remaining models are not starved; they resume per worker as that worker finishes the other models' work (899 resumed on 9 finished workers on 2026-11-01). Attempts for the shared remaining tasks continue.

Attempt evidence

Files become downloadable when the evidence packer has produced them. Paths follow evidence/models/<model>/attempts/<repo-slug>/<instance_id>.json.gz. Repository slugs use the SWE-bench double-underscore form. Evidence and verification.