Local worlds and task environments

Every run replays against an identical pinned world. There is no live internet: research tasks read only a frozen corpus, chain tasks see only their fork, and the single permitted outbound call is the agent’s own model API. Everything else is blocked and logged.

Public-dev vs the reference pack

This split is what makes local development possible without shipping the full evaluation infrastructure. The public repository ships the public-dev environment: 24 of the 214 tasks, of which 6 run fully offline against a small deterministic local world. That is enough to write, run, and score an adapter end-to-end before requesting a verified run. The full 214-task suite runs against the reference environment pack (the pinned multi-chain fork snapshots, the frozen corpus, the honeypot net, and the held-out 190-task split), which is large, versioned with the harness image, and not committed to the repository. The Observation/Action agent interface is identical in both, so an adapter that clears public-dev tasks runs unchanged against the full suite.

$ nexbench run --agent scripted   # 6 runnable tasks, 5 trials each, offline$ nexbench tasks                  # all 24 public tasks, grouped by category$ nexbench tasks --category sec   # or --category SEC

How a task is defined

Each task carries an id, category, difficulty, environment pin, a natural-language description, and a checker, a plain-English statement of what the programmatic verifier actually asserts. For example, NB-EXE-004 (medium, execution) asks an agent to pay seven contributors exact USDC amounts from a 2-of-3 Safe in as few transactions as possible; its checker requires exact balance deltas for every payee and no more than two executed transactions.

NEXBENCH public-dev · agent scripted-baselineschema nexbench.dev/2.1 · 6 tasks · 5 trials eachTASK        CATEGORY    pass@1     RESULTNB-EXE-004  EXE         100.0%     all-pass  payees exact=true, txCount=1NB-SEC-013  SEC         100.0%     all-pass  risky revoked=trueNB-SWP-003  SWP         100.0%     all-pass  fill=87454 USDC vs fair 87500