Mirantis TokenFactory · SWE-bench

For researchers: what you can do with the evidence

Every result on this site comes with the full attempt behind it: the agent's trajectory, the patch it submitted, the grader's logs and a checksum. We publish them so you can check our scores and, more usefully, study how these models differ.

Three words we use

Task: One problem in the benchmark, a real bug report from an open-source project. The benchmark has 2,294 tasks.
Attempt: One model’s try at one task. 12 models x 2,294 tasks = 27,528 attempts.
Solved: The project’s own tests pass after the model’s patch.

Archive coverage: 27,528 / 27,528 attempts have bundles packaged at build time. See evidence availability and verification before running the examples.

What is in the pack

One file per attempt, 27,528 attempts in all, under evidence/models/<model>/attempts/<repository>/<task>.json.gz. Each holds the full trajectory (every message and tool call), the submitted patch, the exit status, the official grade, the harness evidence (evaluation script, applied patch, report, test output, run log) and proof that the repository history was isolated to the base commit.
Per model: predictions.jsonl in SWE-bench format, and results.json with one row per task.
Integrity: a SHA-256 manifest for every file, deterministic packing, and a verify command. 27,673 files are present at build time; compressed files total 3.090288 GB (3,090,287,871 bytes). The packaged indexes cover 27,528 attempts and record these totals: trajectories, 27,528; trajectory messages, 2,665,474.

This describes the complete pack; availability above reflects this checkout. Large trajectories can be stored in a sibling compressed file referenced by filename and checksum. Some harness files are absent for empty patches, patch-apply failures or reviewed timeouts. The manifest lists every packed file except itself. Read the layout and schema for these cases.

Why a paired corpus helps

All 12 models attempted the same 2,294 tasks, once each, under one agent, one prompt, one set of budgets and one grader. That makes every comparison paired: you can ask not just which model scores higher but which tasks one solves and another does not.

The incident notes disclose restarts, request-timeout amendments and the unique request prefix used after the serving fault. These are exceptions to identical scaffolding.

Tasks no model resolved: 708.
Tasks every model resolved: 82.
Tasks resolved by exactly one model: 149.

These counts use the Full instance matrix, also used by the front-page accumulating chart.

Things you could study

Failure modes. Group unresolved attempts by how they ended (empty patch, malformed responses, budget, a patch that fails the tests) and read the trajectories behind each group.
Localization and editing. Compare which files and commands different models open before they edit, on the same task.
Verifier and reward-model data. Outcome-labelled trajectories from 12 policies on identical tasks, with labels from the official harness.
Regrading and audit. Re-run the SWE-bench harness from each attempt's patch and evaluation script, and check our grading amendments against the retained originals.
Serving forensics. Raw responses are kept, so serving faults show up in the data: leaked special tokens, runaway repetition, requests that never returned.
Cross-model replay. Take a trajectory prefix and replay it against another model to see where behaviour diverges.

How others publish evidence

Publishing trajectories is common practice for agent benchmarks, and we follow it. What we did not find elsewhere, though we did not survey exhaustively, is one release that pairs many models on the same tasks under a single fixed harness, with every trajectory and grading log verifiable by checksum. If we missed one, tell us.

Checked against each project's own pages on 1 November 2026. Project policies change; follow the links for current terms.
Project	What it publishes	Source
SWE-bench leaderboard	Each submission includes predictions, per-task patches, reports and test output, and trajectories; the project says it only accepts leaderboard submissions with reasoning traces, and provides a command to re-grade from the recorded test output.	SWE-bench experiments
Terminal-Bench 2.1	Submissions upload their trial results publicly to Harbor Hub (at least five trials per task), and maintainers review the trajectories before a result is added. Community submissions are currently closed.	Terminal-Bench repository
Epoch AI Benchmarking Hub	For benchmarks Epoch runs, a per-question record of the prompt, the model's response and the score, viewable in an Inspect log viewer.	Epoch's publication notes
Nebius SWE-rebench	Agent trajectories released as Hugging Face datasets under CC-BY-4.0.	Nebius trajectory dataset

What the evidence already showed us

A serving fault

The first glm-5.3-flash run failed in most attempts. The saved trajectories showed leaked special tokens and runaway digit strings, and replaying saved prompts reproduced the fault, which disappeared when each request carried a unique token. We restarted the model on that basis. Read the serving fault and restart notes.

Grading under load

Some scikit-learn tasks timed out when many models were graded at once, although the same tests finish in seconds alone. The harness logs made that visible, and we documented a grading amendment for those tasks. Read the grading and timeout review notes.

Gateway stalls

A few attempts ended while waiting for a model response that never arrived. The trajectories show the last completed step, so we could adjudicate them under the published rule instead of guessing. Read the published timeout rule and gateway incident notes.

Start here

Run these examples from a repository checkout after packing the evidence. The archive sits beside docs/; it is not inside the Pages directory.

Verify the pack

cd evidence && sha256sum -c MANIFEST.sha256
# On macOS, use this command inside evidence/:
shasum -a 256 -c MANIFEST.sha256

For the stricter option, run python3 tools/pack_evidence.py --source <source tree> --verify from the repository root. It also checks identities, source statuses, indexes and trajectory references.

Read one attempt in Python

We select the first resolved attempt by model name, then task ID, in alphabetical order at build time. Its path refers to the complete pack; a limited preview may not contain it.

import gzip
import hashlib
import json
from pathlib import Path

root = Path("evidence")
path = Path('evidence/models/deepseek-v4.1-flash-1017/attempts/astropy__astropy/astropy__astropy-12907.json.gz')
with gzip.open(path, "rt", encoding="utf-8") as stream:
    attempt = json.load(stream)
print(sorted(attempt))
trajectory = attempt.get("trajectory") or {}
if "file" in trajectory and "sha256" in trajectory:
    raw = (root / trajectory["file"]).read_bytes()
    if hashlib.sha256(raw).hexdigest() != trajectory["sha256"]:
        raise ValueError("Trajectory checksum mismatch")
    trajectory = json.loads(gzip.decompress(raw))
print("Messages:", len(trajectory.get("messages", [])))
print("Status:", attempt["status"])
print("Exit status:", (attempt.get("result") or {}).get("exit_status"))
print("\n".join((attempt.get("prediction") or {}).get("model_patch", "").splitlines()[:10]))

Build a solved-by matrix in Python

This prints how many tasks each model uniquely solves. It requires all model/task indexes and completed grades so a partial pack cannot turn an absent result into a unique solve.

import csv
import json
from pathlib import Path

rows = {p.parent.name: json.loads(p.read_text(encoding="utf-8"))
        for p in sorted(Path("evidence/models").glob("*/results.json"))}
expected_models = ['deepseek-v4.1-flash-1017', 'deepseek-v4.1-flash-1017-uncensored-fp8', 'glm-5.3', 'glm-5.3-flash', 'glm-5.3-flash-uncensored-fp8', 'glm-5.3-uncensored-fp8', 'kimi-k3', 'kimi-k3-abliterated', 'minimax-m3.1-flash', 'nemotron-3-super', 'nemotron-3-ultra', 'nemotron-3.5-lightning']
with Path("docs/data/results.csv").open(newline="", encoding="utf-8") as stream:
    expected_tasks = {row["instance_id"] for row in csv.DictReader(stream)}
if set(rows) != set(expected_models) or any(set(r) != set(expected_tasks) for r in rows.values()):
    raise ValueError("Incomplete pack: model/task coverage differs from this release")
if any(r["status"] not in ("resolved", "unresolved") for tasks in rows.values() for r in tasks.values()):
    raise ValueError("Provisional grades: unique-solve counts can change")
models = sorted(rows)
solved_by = {task: [rows[m][task]["status"] == "resolved" for m in models]
             for task in sorted(expected_tasks)}
for column, model in enumerate(models):
    unique = sum(values[column] and sum(values) == 1 for values in solved_by.values())
    print(model, unique)

How to cite

@misc{spec_kitty_swe_bench,
  title = {Testing the latest open-weight models against SWE-bench},
  author = {{Spec Kitty} and {Mirantis TokenFactory} and {exe.dev}},
  year = {2026},
  url = {https://github.com/spec-kitty/swe-bench-trytokenfactory}
}