Run it. Submit it. Get ranked. The harness is open source — any agent that can speak five actions over JSON-RPC can be scored.
Quickstart
Four steps from zero to a verified entry. Develop against the public split locally; the held-out evaluation runs on Nexis infrastructure when you submit.
One package: the CLI, the forked-chain environments (via Docker), and the public dev split.
$ npm install -g nexbench$ nexbench init my-agent && cd my-agentOne function — or one HTTP endpoint. The harness drives the observation/action loop and owns all chain access.
# agent.yaml — everything nexbench needs to run your agent id: my-agent adapter: ./adapter.ts # or endpoint: http://localhost:8700/step model: any # bring your own — API or self-hosted networks: [evm, solana] timeout_s: 900 # per-task wall-clock cap cost_cap_usd: 10 # per-task spend cap
// adapter.ts — one function; the harness drives the loop
import type { Observation, Action } from 'nexbench';
export default async function step(obs: Observation): Promise<Action> {
// obs: task brief, wallet + chain state, prior tool results
// Action: rpc_call | sign_request | corpus_query | note | submit
return plan(obs);
}Every task runs 5 trials in a fresh container against its pinned fork. Scores, traces, and a hash-sealed manifest land in ./runs.
$ nexbench run --agent ./agent.yaml --trials 5$ nexbench reportThe CLI mints a run manifest — scores on the trial grid, a content-hashed runId, and your trace-archive root. Validate and submit it on the submit page; verified tier re-executes your image on the private split.
$ nexbench validate ./my-run.json && nexbench submit ./my-run.json{
"schema": "nexbench.run/2.1",
"suite": { "name": "NEXBENCH", "version": "2.1",
"totalTasks": 214, "trialsPerTask": 5 },
"agent": { "id": "my-agent", "name": "My Agent", "scaffold": "custom",
"model": "gpt-5.5", "class": "agent", "openSource": true },
"submitter": { "name": "You", "contact": "you@lab.xyz", "github": "you" },
"run": { "completedAt": "2026-07-11", "harnessVersion": "2.1.3",
"harnessBuild": "sha256:…", "envPinsDigest": "sha256:…" },
"results": {
"categories": { "execution": { "passAt1": 71.875 }, "…": {} },
"metrics": { "passHat5": 48.5981, "svrPer100": 3.7383,
"gasOverspendPct": 3.8, "costPerTaskUsd": 1.18,
"medianTaskSeconds": 141 }
},
"integrity": { "runId": "nbr1_…", "traceRoot": "sha256:…",
"canaryClean": true },
"provenance": { "kind": "community", "tier": "self-reported" }
}Scoring
The math is public and boring on purpose. No judges, no vibes — every number on the leaderboard reduces to these six lines.
FIG.G1
Scoring & statistics
task score s_t = (1/k) · Σᵢ pass(t,i) k = 5 trials category S_c = 100 · mean( s_t : t ∈ c ) overall S = Σ_c w_c·S_c / Σ_c w_c w_c = task count of c interval CI95 = 1.96 · √( p(1−p) · DEFF / N ) task-level bootstrap reliability pass^5 = 100 · mean( Πᵢ pass(t,i) ) safety SVR = 100 · violations / tasks hard fail ⇒ s_t = 0
DEFF < 1 is the design effect from trial averaging (ICC ≈ 0.35 across the suite) · Bootstrap: 10,000 task-level resamples, stratified by category · Ties: two entries whose 95% intervals overlap are reported as statistically indistinguishable.
Verification
Three tiers. One badge that matters. Self-reported numbers are listed but never ranked above a verified run with overlapping intervals.
Verified
Your container re-executed by Nexis on the private split. Score, CI, SVR, and full traces published. Required for official rank claims.
Self-reported
Public-split results submitted with manifest and traces, not re-executed. Listed with a grey badge and excluded from “top agent” statistics.
Reference
Protocol runs operated by Nexis: the human expert panel and the scripted no-LLM baseline. Unranked context rows on every board.
Rules
Six rules keep runs comparable. They exist so a score means the same thing for every agent on the board.
Determinism
Pinned forks, frozen oracles, fixed seeds. The harness owns all entropy — a re-run of your image must reproduce your score within the reported interval.
Isolation
No live internet. Research tasks read the frozen corpus; chain tasks see only their fork. Outbound calls other than your model API are blocked and logged.
No task-specific code
General policies only. Hardcoding task ids, gold answers, or checker logic disqualifies the run — checkers and the private split rotate, so it also doesn’t work.
Budgets
Fifteen minutes wall-clock and $10 model spend per task — generous headroom above every listed agent. Exceeding either fails the trial.
Full trace
Every tool call, signed payload, and token count is written to the run manifest. Verified runs publish their traces alongside the score.
Canary
Every task file embeds a canary GUID. If it surfaces in model output, the run is flagged for contamination review and the task version is retired.
FAQ
Common questions.
Can I benchmark a closed, proprietary agent?
Yes. Point the adapter at an HTTP endpoint — weights and prompts never leave your infrastructure. The verified tier needs a runnable image or an endpoint held stable while we re-execute.
What does a full run cost?
The harness is free and open source; you pay your own inference. Across listed agents a full 214-task, 5-trial run ranged from roughly $130 (small models) to $3,300 (frontier models).
Why programmatic verifiers instead of LLM judges?
Determinism and no self-preference bias. Every checker asserts on post-run chain state, balances, event logs, or gold numeric answers, and is unit-tested against known-good and known-bad traces.
What is the difference between pass@1 and pass^5?
pass@1 is the mean success over five independent trials — expected performance. pass^5 requires all five trials to succeed — reliability. Production agents should be judged on both.
How is training contamination handled?
24 tasks are public for development; 190 stay private and rotate quarterly. Canary GUIDs detect leakage, and suite versions are pinned so old scores stay comparable.
How are fabricated or duplicated results caught?
Twelve intake checks run on every manifest: scores must sit on the mathematically achievable trial grid, the runId content hash must recompute, trace roots and score vectors are collision-checked against every listed run, and unknown harness builds are capped at self-reported. The full list is enforced — and shown live — on the submit page.
Can I submit several configurations?
One leaderboard entry per agent + model + scaffold configuration. Ablations are welcome as separate, clearly named entries.
Changelog
Versioned like software, because it is.
v2.1
2026-06-02
- ·Suite grown to 214 tasks; Solana and Sui forked environments added
- ·Safety hard-fails with the SVR metric reported beside every score
- ·pass^5 reliability metric; gas-oracle replay for fee realism
- ·Public manifest schema (nexbench.run/2.1) + validated submission intake with duplicate and grid checks
- ·Private split rotated (Q2 → Q3 pool); canary GUIDs re-issued
v2.0
2026-02-17
- ·Programmatic verifiers replaced rubric judges across all categories
- ·Deterministic mainnet forks replaced testnets; frozen research corpus
- ·Eight-category taxonomy with task-count weighting
v1.0
2025-09-30
- ·Initial release: 92 tasks, testnet execution, judge-scored research
Benchmarks are how the field moves. Run the suite tonight; be on the board this week.