CyberNativeOpen Research Lab Evidence report

Draft technical report / 001Published 2026-07-21

Seed sensitivity in a tiny neural network

One Good Run
Is Not the Result

With the model, synthetic data, split, and training protocol fixed, held-out accuracy still moved from 91.67% to 95.00% across 12 declared random seeds. This report shows every retained row and the exact evidence behind that narrow result.

Draft report Executed CI artifact verified Not human-approved

Read this first

Executed evidence is not an approved research release.

The recorded experiment ran and its retained artifact passed deterministic integrity checks. That supports inspection of this bounded result; it does not make the result generally true.

Human editorial approval
Not granted
Independent reproduction
Not completed
Authenticated or peer review
Not completed
DOI or permanent archive
None

Observed result

Two predictions separate the lowest and highest runs.

12

declared seedsall retained below

93.06%

mean accuracy56 of 60 is 93.33%

0.96 pp

sample deviationaccuracy across seeds

3.33 pp

full range55/60 through 57/60

Twelve seed results range from 91.67 to 95.00 percent test accuracy; the mean is 93.06 percent.
FIGURE 01 / RETAINED TRIAL A

The vector figure is byte-identical to the generated input retained by the exact Quarto CI run. It is derived from the admitted trial manifest, not copied from this prose.

What changed?

Only the declared seed changed. It jointly affected parameter initialization and training-example order. The selected best run, seed 71 at 95.00%, sits 1.94 percentage points above the 12-seed mean. A report containing only that run would hide the observed spread.

Accuracy is discrete here because the held-out set has 60 examples: three seeds scored 55/60, eight scored 56/60, and one scored 57/60. Seed 29 has the highest test loss despite tying two other seeds on 55/60 accuracy, showing that a thresholded score can also hide confidence differences.

Complete retained rows

No cherry-picked seed table.

Trial A from bundle b49c7f938842716358fe3860395cc5e7f1432f707533c55ba8438de2614f9017.

Test metrics for every seed in the frozen protocol, ordered as declared.
SeedTest accuracyCorrect / 60Test loss
1193.33%56/600.1417
2991.67%55/600.2799
4793.33%56/600.1949
7195.00%57/600.1492
10193.33%56/600.1749
13191.67%55/600.1907
16793.33%56/600.1522
19993.33%56/600.1534
23993.33%56/600.1281
28191.67%55/600.1476
33793.33%56/600.1472
39793.33%56/600.1593

Method

One fixed, zero-cost synthetic protocol.

Question

How much does initialization and training order change test accuracy for one fixed small-data neural-network protocol?

Run design

  • Tiny two-layer multilayer perceptron trained for 35 epochs.
  • Fixed synthetic two-moons classification data: 180 training and 60 held-out examples.
  • Training-set statistics only for standardization; one fixed split and fixed hyperparameters.
  • 12 seeds jointly changing initialization and training-example order.
  • No downloaded data, no personal data, declared network forbidden, and a $0 cost ceiling.

Frozen criterion

Observed accuracy range >= 1.67%. The observed 3.33% range passes, but the protocol was frozen retrospectively. This is not preregistered confirmation.

Interpretation

A useful teaching result, not a general neural-network claim.

Supported

Report the declared distribution.

For this fixed tiny-MLP protocol, selecting one seed could yield 91.67% through 95.00% held-out accuracy. Showing all 12 declared runs communicates more than showing one selected run.

Not supported

Generalization, production readiness, or scientific novelty.

This capsule cannot say whether 12 seeds are sufficient elsewhere, which source of randomness caused each change, or how production models behave. It is a retrospective synthetic machinery canary.

Reproduce locally

Run the capsule; keep the retained evidence intact.

The commands perform two complete trials and verify the resulting bundle. A matching same-environment repeat is not an independent reproduction.

uv sync --frozen
uv run cnlab validate experiments/seed-sensitivity/capsule.json
uv run cnlab repeat experiments/seed-sensitivity/capsule.json --output .lab/seed-sensitivity
uv run cnlab verify .lab/seed-sensitivity

Exact evidence

The public page and the retained CI artifact are separate.

This CSP-compatible page restates checked values for the web. The files below are untouched outputs from the exact successful main workflow.

Quarto HTML artifact1,905,443 bytes · SHA-256 032fa2c5…e0617 Render verification receiptQuarto 1.9.38 · two identical renders Digest-bound input manifestSHA-256 f51e9f28…774f7 Exact main workflowSource b1af62a5…71b96 · completed successfully

LimitationsDo not erase

What remains
unproven.

  • 01
    No preregistration

    The protocol and criterion were frozen retrospectively.

  • 02
    No independent reproduction

    The retained repeats used the same harness and environment.

  • 03
    No hostile-input sandbox claim

    The report renderer consumed trusted repository-controlled material.

  • 04
    No human-approved release

    Agent contributions and checks do not replace an accountable human editor.