Non-endorsement or false belief? A three-margin diagnostic for keyed knowledge scores

A keyed knowledge item usually offers a "not sure" option, and conventional scoring counts the keyed answer as one and assigns zero to abstentions and wrong answers alike. The score therefore merges two events with different social distributions: a respondent who endorsed a false belief, and a respondent who endorsed no belief. This deposit contains a five-step diagnostic for what such a score aggregates — code three response margins (correct, abstention, distractor); estimate each margin separately; decompose the gap under an explicit accounting identity; separate the quantities identified from the observed responses from the one that rests on a conversion assumption, and report that assumption's range; and check item dependence — together with the full analysis code, stored results, and a verification script that re-checks every table cell and principal estimate against a stored result. The diagnostic is applied to a two-item AI knowledge index in Pew Research Center's American Trends Panel Wave 127 (N = 5,101; fielded 15–21 May 2023). Of incorrect answers, 86.5% and 77.7% are abstentions. The share of the correct-answer gap attributable to abstention is 70.1–91.7% for the age, gender, education and income contrasts under comparison-rate conversion and 52.6–80.5% under the own rate; for Black versus White it is 35.2%. A simulation with a known truth recovers a generating share of 100% at 100.0–100.1%; a second simulation with unequal accuracy among attempters shows the conversion assumption alone can change the reported share by a factor of 1.39 to 2.00. A cross-construct check on seven non-AI knowledge items returns computable shares throughout. Enumerating every subset of the seven non-AI items, the mean share falls from 56.5% at one item to 46.0% at seven while the spread across subsets within a contrast falls from 43.5 to 7.9 percentage points; the two-item AI index has only one possible composition and so has no estimable item-selection margin, which the manuscript states. Contents. Manuscript (PDF and DOCX); cover letter; analysis scripts; stored JSON results for every reported number; figures; verification suite; statistical report; workflow capability audit; revision history; and the venue requirements and literature records used. Data. Pew Research Center's American Trends Panel Wave 127 is public and is not redistributed here. No individual-level record is included. The analysis reads the public file directly from the path recorded in each script. Prior version. An earlier, substantially revised version of this analysis is archived at https://doi.org/10.5281/zenodo.24103711. The present version re-bases the claim, changes the framing and title, and adds the item-selection and conversion-sensitivity analyses. Licence. Code and documentation released under the MIT licence. The manuscript text is the authors' own work and is deposited for review and reproducibility purposes. Use of AI tools. AI tools assisted with literature search, drafting, the design of the analysis scripts, and figure generation. Every reported value is computed by a script from public data and re-checked against a stored result; no value was generated. No AI tool is an author.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24296275
Primary Topic
Psychometric Methodologies and Testing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Non-endorsement or false belief? A three-margin diagnostic for keyed knowledge scores

Kexin Wang, Hang Yang, Jiamin Li, Lei Sun
Zenodo (CERN European Organization for Nuclear Research)
Psychometric Methodologies and Testing
article

Non-endorsement or false belief? A three-margin diagnostic for keyed knowledge scores

Kexin Wang, Hang Yang, Jiamin Li, Lei Sun
article en

Abstract

A keyed knowledge item usually offers a "not sure" option, and conventional scoring counts the keyed answer as one and assigns zero to abstentions and wrong answers alike. The score therefore merges two events with different social distributions: a respondent who endorsed a false belief, and a respondent who endorsed no belief. This deposit contains a five-step diagnostic for what such a score aggregates — code three response margins (correct, abstention, distractor); estimate each margin separately; decompose the gap under an explicit accounting identity; separate the quantities identified from the observed responses from the one that rests on a conversion assumption, and report that assumption's range; and check item dependence — together with the full analysis code, stored results, and a verification script that re-checks every table cell and principal estimate against a stored result. The diagnostic is applied to a two-item AI knowledge index in Pew Research Center's American Trends Panel Wave 127 (N = 5,101; fielded 15–21 May 2023). Of incorrect answers, 86.5% and 77.7% are abstentions. The share of the correct-answer gap attributable to abstention is 70.1–91.7% for the age, gender, education and income contrasts under comparison-rate conversion and 52.6–80.5% under the own rate; for Black versus White it is 35.2%. A simulation with a known truth recovers a generating share of 100% at 100.0–100.1%; a second simulation with unequal accuracy among attempters shows the conversion assumption alone can change the reported share by a factor of 1.39 to 2.00. A cross-construct check on seven non-AI knowledge items returns computable shares throughout. Enumerating every subset of the seven non-AI items, the mean share falls from 56.5% at one item to 46.0% at seven while the spread across subsets within a contrast falls from 43.5 to 7.9 percentage points; the two-item AI index has only one possible composition and so has no estimable item-selection margin, which the manuscript states. Contents. Manuscript (PDF and DOCX); cover letter; analysis scripts; stored JSON results for every reported number; figures; verification suite; statistical report; workflow capability audit; revision history; and the venue requirements and literature records used. Data. Pew Research Center's American Trends Panel Wave 127 is public and is not redistributed here. No individual-level record is included. The analysis reads the public file directly from the path recorded in each script. Prior version. An earlier, substantially revised version of this analysis is archived at https://doi.org/10.5281/zenodo.24103711. The present version re-bases the claim, changes the framing and title, and adds the item-selection and conversion-sensitivity analyses. Licence. Code and documentation released under the MIT licence. The manuscript text is the authors' own work and is deposited for review and reproducibility purposes. Use of AI tools. AI tools assisted with literature search, drafting, the design of the analysis scripts, and figure generation. Every reported value is computed by a script from public data and re-checked against a stored result; no value was generated. No AI tool is an author.

Zenodo (CERN European Organization for Nuclear Research)
Jinan University (CN)
Psychometric Methodologies and Testing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon