Bottom line · from claim 1 of 6: The paper's own evidence supports the main result. We did not rerun it.

Claims, one verdict each

Each claim is quoted from the paper or its repository. Claim 1 is the main result; the bottom line comes from its verdict only. Open “Evidence” for the numbers, code and log lines behind each verdict.

1. 01○Supported: Average lenient accuracy

   Re-analysis of shipped outputs

   Paper: Across all experiments, the average lenient accuracy was 61.2%, with a low overall ambiguity rate (<2%).

   Abstract, Results; Section 3.1

   The paper prints all 50 per-condition values in Figure 2 (10 models, 5 prompting strategies). Their mean is 61.23% lenient accuracy, 48.05% strict accuracy and 1.89% ambiguity. These match the paper's 61.2%, 48.0% and "under 2%".

   Evidence: lenient accuracy (%) for each model and prompt, as printed in Figure 2

   Columns use the paper's names: S1 no role, S2 public-health expert, S3 respiratory specialist, S4 public-health expert + source/date, S5 respiratory specialist + source/date.

   Model	S1	S2	S3	S4	S5
   GPT-4o	52.8	53.3	43.3	47.2	41.1
   DeepSeek-R1	57.2	64.5	60.8	58.9	56.9
   DeepSeek-V3	58.9	63.9	58.4	59.4	50.6
   Doubao-Seed	77.7	82.7	76.1	73.4	71.2
   Gemini-2.5	69.7	81.1	73.0	67.0	63.9
   GLM-4 AirX	85.5	70.8	66.4	58.4	52.5
   Llama-4	43.6	50.9	52.0	41.6	40.8
   Mistral-L	41.9	53.1	54.1	44.2	39.2
   Qwen3-235B	51.2	61.7	54.5	58.4	53.3
   Qwen3-Thinking	90.0	89.1	84.5	81.1	79.5

   Mean of the 50 values: 61.23%. As a check on our reading of the figure, in every one of the 50 conditions lenient accuracy, error rate and ambiguity rate add up to 100.0% (±0.1), as they must, because the five labels split into these three groups.

2. 02○Supported: Best configuration

   Re-analysis of shipped outputs

   Paper: in the best configuration, ≈ 84% strict and ≈ 90% lenient accuracy on this all-misinformation benchmark.

   Abstract, Results; Section 5

   The highest lenient accuracy in Figure 2 is 90.0% (Qwen3-235B-Thinking, no role prompt), with 84.2% strict accuracy in the same condition. The highest strict accuracy anywhere is 86.2% (same model, public-health expert prompt), with 89.1% lenient.

3. 03○Supported: Conservative models accept over half

   Re-analysis of shipped outputs

   Paper: more conservative models incorrectly accepted over 50% of false posts.

   Abstract, Results

   In 9 of the 50 conditions the error rate (posts labelled likely true or true) is above 50%. They belong to 3 models: GPT-4o (up to 56.9%), Llama-4 Maverick (up to 56.9%) and Mistral-Large (up to 58.3%).

4. 04○Supported, lenient chance level is 40%: Above chance

   Checked derivation

   Paper: Q1 (Feasibility): Do off-the-shelf LLMs exceed a uniform-guessing baseline (20% over five options) on Chinese COVID-19 misinformation?

   Section 1, question Q1; Section 3.1

   Under the paper's own definitions (Section 2.4), a uniform guess over the five labels scores 20% strict accuracy but 40% lenient accuracy, because lenient accuracy credits two of the five labels. The averages, 48.0% strict and 61.2% lenient, are above both levels. One condition is not: Mistral-Large with S5 (respiratory specialist + source/date), at 39.2% lenient.

   Every post in the benchmark is misinformation (Section 2.1), so an answer of "False" to every post would score 100%. Here, above-chance accuracy shows how often models flag misinformation. It cannot show how well they tell misinformation apart from true posts, because the benchmark has none. The paper itself describes the benchmark as all misinformation.

5. 05!Minor defect: Table 2 description

   Checked derivation

   Paper: Table 2 compares S1 (no-role) vs. S5 (respiratory-specialist + source/date) within each model, reporting percentage-point changes in lenient accuracy and ambiguity.

   Section 3.3 and Table 2

   Table 2 reports composite scores and ambiguity rates for S1 and S5, as its caption says, not percentage-point changes in lenient accuracy. The table's values agree with Figure 2: all 10 pairs of ambiguity rates match the figure exactly, and all 20 composite scores fall inside the range that the figure's strict, lenient and error rates allow.

   To change this verdict: correct the sentence, or add the lenient-accuracy changes to the table.

   Evidence: how the composite scores were checked

   The composite score (Section 2.4) is (2·N1 + N2 − N4 − 2·N5) / N. Figure 2 gives N1 (strict), N1 + N2 (lenient) and N4 + N5 (error), but not how errors split between labels 4 and 5. So each score must fall between (2·strict + (lenient − strict) − 2·error) / 100 and (2·strict + (lenient − strict) − error) / 100. All 20 scores in Table 2 do. For example, GPT-4o with no role prompt: 0.133 falls in [−0.082, 0.349].

6. 06–Not checked: Reproducible release

   Paper: (4) A reproducible release (prompts, code, judging templates, redacted logs) to support Chinese-language infodemic monitoring and future replication.

   Abstract, Contributions; Section 2.5

   The paper does not give a link to the release. The venue page lists a supplementary file. Its download is behind an automated browser check, so we did not open it.

   To change this verdict: send us a public link to the release, or tell us if it is the supplementary file.

What we did not check

These limits carry the same weight as the findings above.

• We did not rerun anything. The experiments queried commercial model services, and we did not query any model again.
• The figure values are our reading. We read the 200 values in Figure 2 by eye. The sum check above tests our reading, not the paper's underlying counts.
• True posts. The benchmark has no true posts, so nothing here measures how often a model wrongly flags a true post.
• The judge. In the paper, one model maps each answer to one of the five labels. We did not check how often it agrees with human labelling.
• Sections 3.4 to 3.6 and Figures 3 and 4 (persona comparison, box plots, trend over time) were not checked.
• The supplementary file was not opened, because its download is behind an automated browser check.
• The rest of the paper. This report checks the claims listed above, looks up every entry in the reference list and reruns code only where it says so. It is not a full review of the paper. For comments on the whole paper, you can use a general AI review tool yourself; Google's Deep Review (https://research.google/deepreview/) offers free weekly reviews with a Google account.

How we checked

Paper

https://openreview.net/forum?id=zYDXTzql3n (PDF sha256 7134349c257391b4866b44cb47bd72a2a9d7a150cbb477854cb0336b9c020122)

Values

Figure 2 (four panels, 50 conditions each) and Table 2, read from the paper and typed into a table.

Computation

a short script computes the means, maxima, counts, the sum check and the composite-score ranges.

Time spent

about 11 minutes, from starting the claim list to finishing this draft.

Outputs and logs

the typed table, the script and its output are published with this report: outputs.

References

Found

13

Matched automatically

11

Not matched

2

Checked by hand

2. Both are real works our tool missed: a 2017 Council of Europe report on information disorder, and a book on social media and democracy from Princeton University Press.

Could not find

0

Author reply

Changes to this report