Human Oversight During AI-Assisted Documentary Coding: An Empirical Study of Decisions, Interventions, and Traceability

A researcher agrees with an AI auditor that a coded category is unsupported and should be removed. The category is still present in the analysed dataset, and no record explains why. When decisions and data are recorded separately, such cases remain hidden, because neither record shows whether one reached the other. This retrospective study linked the two kinds of records in a completed project in which supervised AI systems coded 11,458 online reviews and 187 complaints about the Singapore–Malaysia confinement nanny market. Selected decisions were traced into the analysed data, and all 276 saved batch files were compared to identify changes that no decision explained. Whether a change could be traced depended on how it was made. Corrections logged by record and field were fully traceable. Recoding was not. Three recoded batches produced 109 changes, and only six of these could be linked to a decision about that record. Thirty-one saved files were also modified later, without their earlier contents being kept. Despite these gaps, none of the counts used in the project’s analysis changed, although only rebuilding them afterwards revealed this. A recorded decision is not evidence that it was carried out. The study proposes records that would provide that evidence, for future projects to test.

Authors

Institutions

Publication Details

Journal
Information
Published
2026-11-19
DOI
https://doi.org/10.3390/info17111177
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Human Oversight During AI-Assisted Documentary Coding: An Empirical Study of Decisions, Interventions, and Traceability

Anusuyah Subbarao, Aaron Aw Teik Hong, Tekenate E. Amah, Catherine Tan Lee Huan
Information
Ethics and Social Impacts of AI
article

Human Oversight During AI-Assisted Documentary Coding: An Empirical Study of Decisions, Interventions, and Traceability

Anusuyah Subbarao, Aaron Aw Teik Hong, Tekenate E. Amah, Catherine Tan Lee Huan
article en

Abstract

A researcher agrees with an AI auditor that a coded category is unsupported and should be removed. The category is still present in the analysed dataset, and no record explains why. When decisions and data are recorded separately, such cases remain hidden, because neither record shows whether one reached the other. This retrospective study linked the two kinds of records in a completed project in which supervised AI systems coded 11,458 online reviews and 187 complaints about the Singapore–Malaysia confinement nanny market. Selected decisions were traced into the analysed data, and all 276 saved batch files were compared to identify changes that no decision explained. Whether a change could be traced depended on how it was made. Corrections logged by record and field were fully traceable. Recoding was not. Three recoded batches produced 109 changes, and only six of these could be linked to a decision about that record. Thirty-one saved files were also modified later, without their earlier contents being kept. Despite these gaps, none of the counts used in the project’s analysis changed, although only rebuilding them afterwards revealed this. A recorded decision is not evidence that it was carried out. The study proposes records that would provide that evidence, for future projects to test.

InformationVol. 17(11)
Multimedia University (MY), University of Technology Malaysia (MY)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon