REPRODUCIBILITY · GGUF QUANTIZATION
Quality and memory reproduced across two runs. Wall-clock latency did not.
We ran the same pinned GGUF quantization and evaluation pipeline twice — two operators, fresh containers, one pinned image — on two small Apache-2.0 models, to see which measured quantities survive a repeat.
CPU-only, no GPU. The latency failure is kept in the result, not smoothed over.
What reproduced
Swipe or use the arrow keys →
| Measurement | Across the two runs | Reproduced |
|---|---|---|
| F16 and Q4_K_M GGUF files | Byte-identical (SHA-256) | Yes |
| Perplexity, Wikitext-2 test (fixed 128 × 512-token budget) | Identical to four decimals, each arm | Yes |
| HellaSwag acc_norm, fixed 200-task subset | Identical, each arm | Yes |
| Peak RSS during the perplexity run | Within 0.13% | Yes |
| Wall-clock model load time | Up to 61% spread | No |
| Prompt and generation throughput | Up to 34% spread | No |
Tolerance was fixed in advance at ±2% for every quantity. Both GGUF files and both evaluation sets carry their SHA-256 in the pinned inputs. The two latency rows are a property of our shared CPU host, not a comparison of the models.
Q4_K_M against F16
Swipe or use the arrow keys →
| Model | Perplexity F16 | Perplexity Q4_K_M | HellaSwag F16 | HellaSwag Q4_K_M | Peak RSS F16 | Peak RSS Q4_K_M |
|---|---|---|---|---|---|---|
| Qwen2.5-0.5B-Instruct | 15.3782 | 15.7842 | 50.50% | 50.50% | 1942.6 MiB | 1402.0 MiB |
| SmolLM2-360M-Instruct | 14.2338 | 14.6453 | 52.50% | 54.50% | 1114.1 MiB | 702.8 MiB |
Run 1 of 2. Perplexity is comparable only within a model, between its two arms. HellaSwag acc_norm is not expected to match between the two models; the question is whether each model’s own numbers repeat.
What this does not show
Two executions of one harness are repeat execution, not independent scientific reproduction: the same code, protocol and tolerance produced both runs. This covers two small models and one quant recipe, not GGUF quantization in general. We make no latency or throughput claim from it — reproducible wall-clock measurement needs exclusive CPU. Everything needed to rerun it is published, including the pinned image digest, model revisions and input hashes.