Mirantis TokenFactory · SWE-bench

kimi-k3-abliterated

38.32% Final

879 tasks solved inside our limits. 2,294 of 2,294 Full attempts graded (100.00%).

Limit-constrained: Cut off on 20% or more of tasks by our limits, so this score understates what the model can do.

Outcome mix

Report statuses partition the Full set. Empty patches and reviewed timeouts overlap unresolved outcomes; they are not extra attempts. Incident reruns are not an outcome category.

By repository

Tokens and steps

Read 2,294 trajectories from 2,294 collected attempt directories. Steps use info.model_stats.api_calls. Tokens sum recorded assistant response usage per attempt; prompt tokens include repeated context. Missing fields remain unknown. Partial attempts and partial usage are included as observed totals.

33 inspected trajectories lack prompt or completion usage for some recorded calls. These totals are lower bounds. Download usage summaries and per-attempt metrics.

Notable events

Gateway-stall timeout attribution is unavailable in the supplied report. We observed 71 outer-process timeout exits and 938 wall-budget exits; 71 outer timeouts have explicit adjudications. These counts do not prove that every timeout was caused by the gateway.

Attempt evidence

Files become downloadable when the evidence packer has produced them. Paths follow evidence/models/<model>/attempts/<repo-slug>/<instance_id>.json.gz. Repository slugs use the SWE-bench double-underscore form. Evidence and verification.