Mirantis TokenFactory · SWE-bench
Who made this
We ran this benchmark with Mirantis TokenFactory and exe.dev. Their gateway, machines and credential integration let us put our protocol to work and bring the results back for you to inspect.
Spec Kitty
We chose the protocol, operated the run and graded predictions with the official SWE-bench harness. We publish the result matrix and trajectories so you can follow a score back to the work that produced it. The evidence page shows what is packaged, and grading coverage shows how much has been graded.
Mirantis TokenFactory
Mirantis TokenFactory gave us one OpenAI-compatible endpoint and one API shape for 12 models, with tool calling across every route. We could keep the same request machinery as we moved between models. When we found serving issues, we reported them with evidence so the gateway team could act. You can get access through the Mirantis TokenFactory portal linked in the footer.
exe.dev
exe.dev gave us named Linux VMs we could reach over plain SSH, resize in place and run with Docker and systemd. Its integration kept our gateway credential off every worker. We retained our queues through resizing and used familiar tools to collect the results. Thank you for making it possible to keep working on the same machines as the run grew.
How we worked together
We at Spec Kitty wrote the scheduling, recovery and grading logic and prepared this publication. Mirantis TokenFactory carried the model requests, and exe.dev ran the machines behind them. Thank you to both teams for the concrete things that made this work possible. You can read how we used them in the infrastructure account and follow the run’s interruptions in the incident notes.
A joint project. Every score here is computed from the public evidence in this repository. For researchers: what you can do with the evidence.