Evaluation datasets for frontier AI.

Tasks are containerised — each comes with a reproducible Docker image, a frozen test suite, and explicit pass/fail criteria. Models are graded both on whether the final state matches expectations and on the trajectory taken to reach it.

Coverage spans data wrangling, environment setup, package management, debugging, and multi-tool orchestration. We add new tasks as new shell-based failure modes appear in the wild.

Authored by olympiad medalists and competition coaches. Problems are deep, not deep-and-wide — each one targets a specific reasoning capability that smaller models tend to skip past.

Every example ships with the canonical solution, a verifiable final answer, and a notes field explaining what the problem is actually testing.

PhDs in their field author problems that draw across multiple research sources, with answers that can be verified against the literature. Designed to defeat surface-level retrieval and lookup heuristics.

Covers domains where the answer is unambiguous if you know the field — and almost guessable if you don't.

Each question is written by a working scientist and validated by a second independent reviewer. Distractors are designed to look right to a model that's just pattern-matching surface vocabulary.

Tightly scoped across physics, chemistry, and biology — with explicit answer rationales and source citations.

Realistic programming tasks across multiple languages. Each task ships with a public test suite, a hidden test suite, edge cases, and where relevant a performance benchmark.

Authored and reviewed by senior engineers — the same people who would grade the output if it landed in a code review at their day job.

Adversarial prompts that probe the seams of a model's safety policy, paired with the policy-correct response. Built to defeat refusal-pattern shortcuts and surface real failure modes.

Calibrated against helpfulness so the resulting models don't over-refuse benign requests — we measure both axes.