Mirantis TokenFactory · SWE-bench

deepseek-v4.1-flash-1017

54.97% Final

1,261 tasks solved inside our limits. 2,294 of 2,294 Full attempts graded (100.00%).

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Outcome mix

Report statuses partition the Full set. Empty patches and reviewed timeouts overlap unresolved outcomes; they are not extra attempts. Incident reruns are not an outcome category.

By repository

Tokens and steps

Read 2,294 trajectories from 2,294 collected attempt directories. Steps use info.model_stats.api_calls. Tokens sum recorded assistant response usage per attempt; prompt tokens include repeated context. Missing fields remain unknown. Partial attempts and partial usage are included as observed totals.

244 inspected trajectories lack prompt or completion usage for some recorded calls. These totals are lower bounds. Download usage summaries and per-attempt metrics.

Notable events

Task-wide grading amendments and reviewed timeout policies apply. See the methodology and each attempt’s evidence.

Attempt evidence

Files become downloadable when the evidence packer has produced them. Paths follow evidence/models/<model>/attempts/<repo-slug>/<instance_id>.json.gz. Repository slugs use the SWE-bench double-underscore form. Evidence and verification.