Secure code review game · 171 pairs

Which one ships?

Two versions of the same function. In a sandbox, an attack broke one of them and failed on the other. Pick the one you would merge, then see the attack and the changed lines. A ruler plays along: it always picks the longer version.

Round 1 of 10You 0Ruler 0

Loading the pairs…

Where the pairs come from

Every pair is a row of the public Code Vulnerability Security DPO v2 preview on Hugging Face, revision bfa102e4: all 171 of them, 96 in Python, 52 in C and 23 in JavaScript. An open-weight model, Qwen3-Coder 30B run on our own hardware, wrote both versions. Our harness, not the model, chose the attack inputs and judged the result, in a sandbox with no network. A pair is in the set only if the attack worked on one version and failed on the other, and a functional test passed on both. Comments were removed from both versions. Some fixed versions still give themselves away with names like safePath.

Why a ruler

DPO, the training method this dataset is made for, can reward length over substance: Park, Rafailov, Ermon and Finn (2024) found significant length exploitation in it on summarization and dialogue. If the fixed version is nearly always the longer one, a model can learn “pick the longer code” instead of “pick the safe code”. In this set the fixed version is the longer one in 131 of 171 pairs (77%), and in 23 of the 23 JavaScript pairs. In v1 of the dataset it was 4,111 of 4,656 (88%). When you beat the ruler, you read the code, not its length.

Limits

  • No human reviewed these pairs. AI reviewers read 79 of the 171, all 52 C pairs and 27 of the Python and JavaScript ones, and labelled every one sound. Where a reviewer left a note, it appears after you choose. The other 92 were checked only by the harness.
  • A pair shows one attack class on one small function. A version that held here can still be wrong in ways the harness did not test.
  • On 16 November 2026, 11 JavaScript pairs were removed from the dataset because their own functional test failed. They are not in the game. The dataset card has the correction.