Alpha Challenge 2026 — Model Leaderboard

Verified score comparison Shared 0–750 scale · zero baseline Evidence paths exhausted Compute exhausted

Leaderboard

Loading verified results…

¹ Why Challenge 06 is excluded: Codex provider safety controls repeatedly blocked attempts involving this challenge. It was then removed from every model’s workspace and from the denominator so all ranked runs faced the same scope.

² Sol safety limitation: some Sol attempts were halted by the provider’s cybersecurity safety filter. Those interruptions are treated as unscored execution limitations—not incorrect answers—and no blocked claim receives points.

³ Calls versus iterations: a tool call is one shell, web, or file action; an iteration is one model turn and may contain zero, one, or several tool calls. Each row labels the retained count by type. When tool-call telemetry was not retained, an iteration count is shown as a proxy rather than relabeled as tool calls. The 200 limit is a ceiling, and models may finish early or reserve final actions for checker runs, trace preservation, and receipts.

⁴ Sol coverage note: Sol verified the bridge-tracing answer, then spent most of its remaining actions exhaustively paging through more than 700,000 densely packed failed Solana attempts for the true first TRUMP snipe. It stopped at 195 actions without freezing or guessing a signature. It is ranked at 25/750; every unattempted or unresolved challenge receives zero.

⁵ What each sandbox included: every model received its own writable copy of the public challenge repository with Challenge 06 removed, standard file and shell tools, the same run instructions, and permission to use approved public internet, RPC, and explorer endpoints. The generic harnesses were Claude Code/Agent SDK (Opus, Sonnet, Fable), Grok CLI (Grok), Codex (Luna, Sol), direct tool loops (DeepSeek, GLM), and Hermes Agent (Kimi, GLM 5.3 Flash (OxAlpha)). “Bare” means no Alpha-specific skill, memory, retrieval corpus, solution scaffold, benchmark-aware helper, or other model’s context was loaded. The benchmark did not guarantee Foundry, forge-std, RPC URLs, API keys, a configured .env, blockchain datasets, or answer hints. Models chose whether to locate, install, or configure those resources. They received no wallet signing keys, real funds, other models’ work, or authority to broadcast transactions; any executed transactions occurred only inside local historical-fork simulations.

Diagonal pinstripes — evidence paths exhausted: the model stopped after concluding that no evidence-backed route remained while substantial portions of both allowances were still available. This applies to Opus (88 minutes, 116 actions) and Luna (70 minutes, 130 calls). Dot pattern — compute exhausted: the run reached its two-hour ceiling or reached/nearly reached its call or iteration ceiling after allowing for checker, trace, and final-receipt actions.

How does the checker know the right answers?

Research questions use a sealed answer key. Wintermute stored only the SHA-256 hash of each normalized correct answer in the public repository. The local checker normalizes a submitted answer, hashes it, and compares the two hashes. It can confirm a match without revealing or downloading the plaintext answer.

Code questions use executable tests. The supplied Foundry test runs the submitted Solidity against the specified historical blockchain fork and awards points only when the required end state is reached.

The checker is local and deterministic: it does not fetch answers from Wintermute. A hash is a one-way fingerprint, so models still have to research the underlying answer rather than recover it from the checker.

Loading analysis…

Permanent benchmark record

Cite this benchmark

Use the frozen v1.0 snapshot for comparisons. Its results will not change as later models are added.

Author: Karina Qian
Version: 1.0 · 27 August 2026
Archive: Frozen release ↗

BibTeX

@misc{qian2026alpha,
  author       = {Karina Qian},
  title        = {When Blockchain Becomes Legible to Models: Wintermute Alpha Challenge 2026 Model Benchmark},
  year         = {2026},
  version      = {1.0},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/ykq/solana-nonvote-indexer/releases/tag/blockchain-alpha-benchmark-v1.0-2026-08-27}},
  note         = {Checker-verified benchmark snapshot, Challenge 06 excluded}
}

Experiment design and reproducibility

Methodology

How the sandboxes were prepared, what each model received, and how runs became checker-verified scores.

Orchestration and sandbox setup

GPT-5.6 Sol served as the experiment orchestrator. Sol prepared a separate writable sandbox for each run, copied the public challenge repository, removed Challenge 06 from the shared scope, inserted the run-specific workspace and iteration allowances, launched or monitored the harnesses, and collected the resulting receipts. Sol did not give one model another model’s answers or solution artifacts.

PrepareCreate an isolated workspace from the same public challenge source and exclude Challenge 06 uniformly.
Run blindGive each model the shared task prompt, public-network access, and its stated time and activity ceilings.
Freeze evidenceRequire evolving Markdown and JSON receipts, then preserve the final answer, code, trace, and run metrics.
ScoreAward points only for checker-confirmed answers or explicit owner adjudication recorded in the benchmark data.

Comparable core

Every ranked model worked independently in a fresh writable copy, attempted challenges 00–05 and 07–08, and could use public RPC, explorer, and web endpoints. Models had no wallet keys, funds, other models’ work, private answer corpus, or authority to broadcast transactions.

Harness differences

The benchmark compares deployed model-plus-harness systems, not bare weights under an identical inference server. Runs used Claude Code or the Agent SDK, Grok CLI, Codex, Hermes Agent, or direct tool loops. Those differences are disclosed beside each model because they can affect tool use and persistence.

Scoring and adjudication

Executable challenges required passing the supplied Foundry checker against the specified historical fork. Research answers were normalized and checked against sealed hashes. Unfinished, unverifiable, or unattempted work scored zero. DeepSeek V4’s Bridge.xyz credit is explicitly owner-adjudicated because the retrieved public label was semantically correct but differed from the checker’s normalized string.

Time, activity, and cost

Elapsed time is wall-clock time when retained. Activity is labeled as tool calls, agent actions, or iterations according to the source telemetry; one is never silently relabeled as another. API costs are equivalent estimates unless an actual charge was retained.

Stopping and retained evidence

Runs could stop before either ceiling when no evidence-backed path remained. Partial discoveries remain visible as retained evidence but do not increase the ranked score. Superseded attempts, traces, strategy records, and receipts are preserved rather than overwritten.

Known limitations

The sample is one public challenge suite and one run configuration per ranked entry. Harnesses, provider safety policies, model pricing, RPC availability, and stochastic model behavior are potential confounders. The results establish performance in these observed runs, not a universal ordering of model capability.

Reproducibility

Prompt used

The isolated workspace path and iteration allowances were filled in for each run. Provider and harness system instructions necessarily varied.

Benchmark task prompt

Independently solve Wintermute Alpha Challenge 2026 in {ISOLATED_WORKSPACE}. Attempt 00-05 and 07-08. Never open or inspect challenge 06; mark it user_skipped. Do not inspect any other /tmp/alpha-2026-* workspace, prior response, branch, history, PWA data, or solution writeup. Work only inside this repository. Public RPCs are allowed. Every HTTP request must use curl with User-Agent exactly 'OpenAI File Downloader, XaiImageApiFetch/1.0'. Freeze analysis answers before checks and never hash-guess. Within three iterations write BOTH MODEL_ATTEMPT.md and valid MODEL_ATTEMPT.json. Keep them current throughout the run: after every solved, failed, or abandoned challenge, immediately record the result, evidence, exact checks, confidence, blockers, and honest point claim. Modify each in-scope Solution.t.sol or answer.txt with your own attempt. You have {MAX_ITERATIONS} tool iterations; the final {FINALIZE_ITERATIONS} are reserved exclusively for validating and completing the two result files.

Wintermute Alpha Challenge 2026