Mirantis TokenFactory · SWE-bench

Read and verify the evidence

Follow a result back to the submitted patch, trajectory and grading records. The scored table and public archive have separate completeness checks.

27,528 / 27,528 expected attempts have bundles present locally. Evidence tree: 27,673 files, 3,156,680,091 bytes (3010.44 MiB). These sizes describe files present at build time, not a promised final download.

For researchers: what you can do with the evidence describes paired comparisons, research questions and examples you can run.

Layout and schema

evidence/
  MANIFEST.sha256
  models/<model>/attempts/<repo-slug>/<instance_id>.json.gz

Repository slugs use the SWE-bench double-underscore convention. Each compressed JSON attempt bundle is intended to preserve prediction, generation result, trajectory, grade, isolation and protocol records, plus available official harness artifacts. Optional evidence includes timeout adjudication and grading history. Empty patches, rejected patches and reviewed timeouts can lack some harness files.

The packer schema includes identity and status, prediction, result, grade, isolation, config_sha256, protocol_amendment, generation and grade logs, trajectory, grading, amendment/adjudication records and grading history. The grading object holds the evaluation script, applied patch, parsed report, test output and run log; absent files are null. Supplemental Verified evidence stays separate in supplemental_verified.

Large trajectories may move to a sibling compressed file; the bundle then references its evidence-relative filename and SHA-256. configs.json deduplicates configurations by canonical JSON hash. Model directories contain predictions.jsonl and results.json indexes. The packer owns the exact versioned schema; the site does not create substitute bundles.

Verify a repository checkout

(cd evidence && sha256sum -c MANIFEST.sha256)
python3 tools/pack_evidence.py --source /path/to/collected-source --verify

Run from the publication repository root after packing. The packer writes evidence-relative manifest entries, so sha256sum -c evidence/MANIFEST.sha256 from the root would look in the wrong directory; change into evidence as shown. On macOS, use shasum -a 256 -c MANIFEST.sha256 there. Packer verification requires the collected source path. These commands verify the archive; the site generator never runs the packer or contacts the fleet.

Download the SHA-256 manifest

Read the evidence packer

Reproduce the analysis

The source project README describes configs/experiment.json for upstream pins, configs/datasets.lock.json for dataset revisions and hashes, and requirements.freeze.txt for the dependency freeze. scripts/snapshot.py records datasets; scripts/preflight.py checks gateway/tool behavior; scripts/isolate_repository.py enforces base-only history. The source project fleet scripts collect the raw artifacts. Those operations require the original environment and are outside this offline publication build.

On an existing collected snapshot, the source project’s scripts/report.py validates grading evidence, scripts/subsets.py handles subset contracts, and scripts/render_report.py renders its simple report. We consume their exports without executing or changing them.

python3 tools/build_site.py --source /path/to/collected-source
python3 -m unittest discover -s tests -v
python3 tools/check_links.py docs
python3 -m http.server --directory .

For a usage preview, add --limit with a positive per-model trajectory count. The score matrix remains complete. Serve the repository root and open /docs/ so relative evidence links also work. For GitHub Pages, build with --publish to link evidence downloads directly to GitHub; the archive is outside the Pages document root. Add --staging for a partner-review preview that asks search engines not to index the pages.

Machine-readable site data

results.csv has model, instance_id, repo and status columns. instances.json stores task rows as compact arrays with model/repository/status dictionaries and availability flags. Model JSON contains safe numeric usage summaries and generation/grade labels; it does not copy raw messages or credentials. comparisons.json contains paired denominators, discordant counts and exact p-values. Build provenance lists source hashes and metric sampling scope.