Bottom line · from claim 1 of 5: The main result holds up when rerun.

Claims, one verdict each

Each claim is quoted from the paper or its repository. Claim 1 is the main result; the bottom line comes from its verdict only. Open “Evidence” for the numbers, code and log lines behind each verdict.

1. 01✓Reproduced: Embedding vs empirical correlation

   Rerun · 1 run

   Paper: Zero-order embedding correlations correlated 0.377 with absolute empirical zero-order correlations. Absolute partial embedding correlations correlated 0.343 with absolute empirical partial correlations.

   Abstract

   Our run of the authors' archive gave 0.37709 and 0.34324, against the paper's 0.377 and 0.343. The archive computes both values once, from the full data. The respondent resampling only produces the ranges around them, so one run is enough to check these two numbers.

   Evidence: paper and our run

   Comparison	Paper	Our run	Difference
   Zero-order, 44,850 pairs	0.377	0.377090	0.00009
   Partial, 44,850 pairs	0.343	0.343238	0.00024

   Where it is computed (analysis/04_compare.R): magnitude_r=cor(abs(scores[[n]]),abs(emp[[kind]])). This is the correlation of absolute embedding values with absolute empirical values over all pairs, with no resampling.

2. 02✓Reproduced: Rank correlations

   Rerun · 1 run

   Paper: The corresponding rank correlations were 0.303 and 0.046, showing that the partial comparison offers little agreement in the overall ordering of pairs.

   Abstract

   Our run gave 0.30306 for the zero-order comparison and 0.04554 for the partial one, against 0.303 and 0.046.

3. 03✓Reproduced: Sample and pair counts

   Rerun · 1 run + Checked derivation

   Paper: Using 7,325 complete records from Johnson's public archive and bootnet for network estimation and respondent resampling, we compare the two across 44,850 item pairs.

   Abstract

   Our run kept 7,325 complete records out of 20,993 in the source file. With 300 items there are 300 × 299 / 2 = 44,850 pairs.

4. 04◐Partly rerun (1 of 2): Both directions

   Rerun · 1 run + Read code

   Paper: Larger absolute embedding correlations accompanied both stronger positive and stronger negative empirical associations.

   Abstract

   For the zero-order comparison this holds. Every embedding cosine is positive (one of the archive's own checks), so larger values are larger absolute values. Among the 23,995 pairs with a positive empirical correlation, the embedding value correlates +0.462 with the empirical one. Among the 20,855 negative pairs it correlates −0.253, so larger embedding values go with stronger negative associations. For the partial comparison, the archive computes the same statistic with signed embedding partials rather than absolute ones, so it does not test the sentence as the abstract words it. We did not compute the absolute version.

   To change this verdict: tell us whether the sentence refers to the signed partial values that the archive uses or to absolute values.

5. 05✓Reproduced: Reproducibility archive

   Rerun · 1 run + Read code

   Paper: The reproducibility archive is publicly available at https://github.com/DennyBorsboom/ipip-neo-300-embeddings-reproducibility. It includes source records, saved embeddings, anonymous item responses, R scripts, all bootstrap estimates, numerical results, and figures.

   "Data and code" paragraph

   At the commit we checked, the archive holds each listed item. Its documented validation command passed all 70 checks. Its full run, using the saved bootstrap results, reproduced every number in the shipped results file to within 0.00000000000003. We used R 4.5.3 rather than the paper's 4.6.0, because one required package had no build for R 4.6 in our package source.

   Evidence: the commands

   Rscript run_all.R --verify     -> All 70 validation checks passed
   Rscript run_all.R --serial     -> exit 0 after 291 s; headline_results.csv matches the shipped file
   

What we did not check

These limits carry the same weight as the findings above.

• We did not redraw the respondent bootstraps. The full run reused the archive's saved bootstrap results, so the ranges around each estimate are the authors', not ours.
• The ranges. The paper prints the resampling range [0.315, 0.324] next to the partial result of 0.343 and describes such ranges as diagnostics, not confidence intervals. We did not check them.
• The embeddings were not regenerated. We used the archive's saved embeddings and did not query the embedding service again.
• Exploratory choices. The paper states that the magnitude comparisons, the sign separation and the curved fits were developed after inspecting the associations. We did not judge that choice, and we did not check the curved fits, the permutation tests or the figures.
• The rest of the paper. This report checks the claims listed above, looks up every entry in the reference list and reruns code only where it says so. It is not a full review of the paper. For comments on the whole paper, you can use a general AI review tool yourself; Google's Deep Review (https://research.google/deepreview/) offers free weekly reviews with a Google account.

How we checked

Repository

https://github.com/DennyBorsboom/ipip-neo-300-embeddings-reproducibility

Commit

0f9d7acf9b2b09cf526ad94b53564ba4ef98c2e8 (archive sha256 58485a68f92022a13feaff3260fccc9e187d2c301298b1038493560c42216571)

Environment

R 4.5.3 with bootnet 1.9.1 and qgraph 1.9.8, as in the archive's session information; contained runner with no network.

Commands

Rscript run_all.R --verify, then Rscript run_all.R --serial.

Seeds

none are needed for the point estimates. The saved bootstrap results come from the authors' own seed file.

Runtime

full run 291 s.

Time spent

writing this draft from the finished rerun outputs took about 2 minutes. This excludes the runs (see Runtime) and the claim list, which were prepared separately.

Outputs and logs

both runs' logs, results files and validation files are published with this report: outputs.

References

Found

13

Matched automatically

7

Not matched

6

Checked by hand

6, all real works our tool missed. Two DOIs broke across a line in our text extraction, and both resolve to the cited papers. Two are web pages that open: the IPIP facet key and the IPIP-NEO data repository. Two are standard book chapters without DOIs: Messick (1989) and Shweder and D'Andrade (1980).

Could not find

0

Author reply

Changes to this report