Reading Is Not Counting: A Three-Model Test of Context Dependence in Expert-Coded Invasion-Biology Studies

Ecological studies can disagree because they test a hypothesis under different conditions. A literature synthesis must identify those conditions without inventing patterns. We test this task against Hi-Knowledge expert coding using 262 sets of eight studies, 299 study pairs and 386 single-study items from held-out papers. Claude Sonnet 5.5 (Anthropic, claude-sonnet-5-5), GPT-6.1 Sol (OpenAI, gpt-6.1-sol) and Muse Spark 1.3 (Meta, muse-spark-1.3-contributor) each receive the same items in five input forms for sets: title, abstract, model-made record, record with expert general conditions, and record with expert general conditions and link information. Pairs use three arms and single studies four. All use medium effort. The original design and a later model-extension plan were frozen before their respective test outputs. Scoring uses the expert coding, with no AI judge. With abstracts and an instruction to tabulate support by condition, Sonnet 5.5 finds 37/103 real patterns and raises 48/118 false alarms; GPT-6.1 Sol finds 58/103 with 99/118 false alarms; Muse Spark 1.3 finds 51/103 with 69/118. The false-alarm rate exceeds the detection rate in every model. Single-study abstract accuracy is similar: 278, 279 and 280 of 386. H2 favours expert conditions over model-made records in all three models, but is significant after Holm only for GPT-6.1 Sol and Muse Spark 1.3; no between-model effect test was run. GPT-6.1 Sol records lower detection (H1) and also lower false alarms (H4: 99 to 87/118; Holm p = 0.0338). Its single-study accuracy drop from 279 to 215/386 is descriptive, not the tested H1 loss. No pre-registered comparison is significant after Holm for Sonnet 5.5. The pattern remains visible in each model's own stances in 50/66, 18/45 and 32/52 abstract-arm misses. Exploratory code counting over those stances with expert conditions finds 83, 67 and 76 of 103 patterns, with 19, 16 and 24 false alarms. This is an upper bound using the scoring rule. The paired detection gain is significant for Sonnet 5.5 and Muse Spark 1.3 but not GPT-6.1 Sol, so the extension's planned all-model criterion is not met. Poor set-level reliability extends beyond the original model, but the balance of reading errors, missed patterns and false alarms differs. Evidence systems need separate checks on study reading, condition extraction and aggregation.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24368430
Primary Topic
Meta-analysis and systematic reviews
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Reading Is Not Counting: A Three-Model Test of Context Dependence in Expert-Coded Invasion-Biology Studies

S. M. Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Meta-analysis and systematic reviews
preprint

Reading Is Not Counting: A Three-Model Test of Context Dependence in Expert-Coded Invasion-Biology Studies

S. M. Abdulla Al Mamun
preprint en

Abstract

Ecological studies can disagree because they test a hypothesis under different conditions. A literature synthesis must identify those conditions without inventing patterns. We test this task against Hi-Knowledge expert coding using 262 sets of eight studies, 299 study pairs and 386 single-study items from held-out papers. Claude Sonnet 5.5 (Anthropic, claude-sonnet-5-5), GPT-6.1 Sol (OpenAI, gpt-6.1-sol) and Muse Spark 1.3 (Meta, muse-spark-1.3-contributor) each receive the same items in five input forms for sets: title, abstract, model-made record, record with expert general conditions, and record with expert general conditions and link information. Pairs use three arms and single studies four. All use medium effort. The original design and a later model-extension plan were frozen before their respective test outputs. Scoring uses the expert coding, with no AI judge. With abstracts and an instruction to tabulate support by condition, Sonnet 5.5 finds 37/103 real patterns and raises 48/118 false alarms; GPT-6.1 Sol finds 58/103 with 99/118 false alarms; Muse Spark 1.3 finds 51/103 with 69/118. The false-alarm rate exceeds the detection rate in every model. Single-study abstract accuracy is similar: 278, 279 and 280 of 386. H2 favours expert conditions over model-made records in all three models, but is significant after Holm only for GPT-6.1 Sol and Muse Spark 1.3; no between-model effect test was run. GPT-6.1 Sol records lower detection (H1) and also lower false alarms (H4: 99 to 87/118; Holm p = 0.0338). Its single-study accuracy drop from 279 to 215/386 is descriptive, not the tested H1 loss. No pre-registered comparison is significant after Holm for Sonnet 5.5. The pattern remains visible in each model's own stances in 50/66, 18/45 and 32/52 abstract-arm misses. Exploratory code counting over those stances with expert conditions finds 83, 67 and 76 of 103 patterns, with 19, 16 and 24 false alarms. This is an upper bound using the scoring rule. The paired detection gain is significant for Sonnet 5.5 and Muse Spark 1.3 but not GPT-6.1 Sol, so the extension's planned all-model criterion is not met. Poor set-level reliability extends beyond the original model, but the balance of reading errors, missed patterns and false alarms differs. Evidence systems need separate checks on study reading, condition extraction and aggregation.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Meta-analysis and systematic reviews
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon