Submitting a run
NEXBENCH3 min read
Mint a manifest, validate it against the shared 12-check intake validator, and submit it to the leaderboard.
A full run against the reference environment pack produces a results draft; nexbench mint turns that draft into a hash-valid manifest by snapping category rates onto the achievable trial grid and computing the run id over the finished results block. You cannot hand-author a valid run id. Minting is the only correct path to one.
- 1
Mint
Assemble a complete manifest from your run draft (agent identity, submitter, category rates, metrics, trace root).
$ nexbench mint --from draft.json --out my-run.json - 2
Validate
Run all twelve intake checks locally, the exact checks the intake API enforces. Exit code 0 means accepted.
$ nexbench validate my-run.json - 3
Submit
Dry run by default: it validates and prints the destination but sends nothing. Add --yes to actually send.
$ nexbench submit my-run.json --yes
nexbench submit refuses to send anything that would be rejected, so a submission never leaves your machine unless it would already be accepted.
One validator, three call sites
The twelve-check catalog is one exported list of checks and one validator function, imported verbatim by the submit portal’s browser preview, the intake API, and this documentation, so a rule added there updates all three at once. Error-severity failures block intake outright; warn-severity flags pass but hold the entry for manual review.
| # | Check | Severity | Rejects |
|---|---|---|---|
| 1 | Schema shape | error | Malformed or mistyped manifests |
| 2 | Suite & environment pins | error | Wrong suite/version/task-count or a modified envPinsDigest |
| 3 | Metric bounds & budgets | error | Rates outside [0,100]; runs over the 900s / $10 per-task budget |
| 4 | Trial-grid alignment | error | Scores off the achievable m/(N·k) grid (catches fabricated numbers) |
| 5 | Internal consistency | error | pass⁵ exceeding weighted pass@1; a wrong category set |
| 6 | Run-id recomputation | error | Any post-mint edit to a covered field |
| 7 | Canary attestation | error | Contaminated runs: the canary GUID surfaced in output |
| 8 | Duplicate run / trace | error | Resubmitted runIds and replayed trace archives |
| 9 | Near-duplicate scores | warn | A score vector within one grid step of an existing entry |
| 10 | Published harness build | warn | Unknown harness images (capped at self-reported) |
| 11 | Submitter identity | error | Missing a contactable email or HTTPS URL |
| 12 | Unexpected fields | warn | Fields outside the schema, a spoofed-badge vector |
Provenance and verification tiers
| Field | Values | Meaning |
|---|---|---|
| Provenance kind | reference · seed · community | Run by Nexis · an illustrative entry · a third-party submission |
| Verification tier | verified · self-reported | Nexis re-executed and reproduced the score · the checks passed but were not re-executed |
Submission rules
One leaderboard entry per agent + model + scaffold configuration. Ablations are welcome as separate, clearly named entries. Hardcoding task ids, gold answers, or checker logic disqualifies a run outright and does not generalize, since checkers and the private split rotate quarterly. Closed, proprietary agents are fine: point the adapter at an HTTP endpoint and your weights and prompts never leave your infrastructure.
Was this page helpful?
Edit on GitHub