From Model Performance to Audit Evidence: An Experimental Study of Purpose-Specific Validity in Algorithmic Audit Procedures

Audit firms validate technological tools before approving them, and the IAASB's proposed ISA 500 (Revised) leaves whether a tool achieves its intended purpose largely to the firm's quality management. This working paper asks to what extent conventional validation results justify relying on a tool's outputs for a specific assessed risk, and what further evidence is needed when performance may not carry over to the engagement. It distinguishes three conditions: the reliability of the input information, the correctness of the computation, and purpose-specific validity. Experiments on a public synthetic general ledger test three implications. Study 1 finds that, for risks the firm's validation covered, relying on an approved tool for a specific risk was unjustified in 84% of decisions under aggregate validation and 2% when validation was organised by risk, with no justified reliance forgone; for risks it did not cover, an engagement-level known-answer test cut unjustified reliance from 67% to 3%. Study 2 finds that effective tools stayed effective under four planned changes while monitoring signals raised false alarms; in an exploratory scenario, a tool refitted each period lost most of a risk type while its exceptions, fixed in number by the review budget, kept their usual spread across accounts. Study 3 finds that known-answer tests estimate detection within 0.05 on average, erring low for tools that rely on rarity, and falsify unfit tools cheaply, but speak only to cases like those seeded; remainder samples of affordable size rarely reveal a wholly missed condition. The rates describe one synthetic generator and one suite of tools; they show mechanisms, not their prevalence in practice. The paper proposes draft application material for ED-500 and ED-330. Not peer reviewed. The studies were specified in advance in a versioned design document with a dated change log; the document was not deposited with a third party before the studies ran. One scenario and two checks were added after results had been seen, and are labelled exploratory or post hoc throughout. The author draws on this study in responses to the AUASB and IAASB consultations on the proposed ISA 330, ISA 500 and ISA 520. How this was made: the author set the research question, designed the studies and made every analytical and editorial decision, and is responsible for the content. Code, analyses and drafts were prepared with Claude (Anthropic) under the author's direction, and each substantive claim was traced to its source in a claims ledger. Data and attribution: the experiments use the DIB General Ledger Data by Gronewald, Stephan, Rombach and Fettke (2026), Zenodo, doi:10.5281/zenodo.22014653, licensed CC BY 4.0, together with an adapted version of its generator notebook, used under CC BY 4.0. The changes are listed in the code, which is available from the author on request.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24155696
Primary Topic
Auditing, Earnings Management, Governance
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

From Model Performance to Audit Evidence: An Experimental Study of Purpose-Specific Validity in Algorithmic Audit Procedures

Maria Prokofieva
Zenodo (CERN European Organization for Nuclear Research)
Auditing, Earnings Management, Governance
preprint

From Model Performance to Audit Evidence: An Experimental Study of Purpose-Specific Validity in Algorithmic Audit Procedures

Maria Prokofieva
preprint en

Abstract

Audit firms validate technological tools before approving them, and the IAASB's proposed ISA 500 (Revised) leaves whether a tool achieves its intended purpose largely to the firm's quality management. This working paper asks to what extent conventional validation results justify relying on a tool's outputs for a specific assessed risk, and what further evidence is needed when performance may not carry over to the engagement. It distinguishes three conditions: the reliability of the input information, the correctness of the computation, and purpose-specific validity. Experiments on a public synthetic general ledger test three implications. Study 1 finds that, for risks the firm's validation covered, relying on an approved tool for a specific risk was unjustified in 84% of decisions under aggregate validation and 2% when validation was organised by risk, with no justified reliance forgone; for risks it did not cover, an engagement-level known-answer test cut unjustified reliance from 67% to 3%. Study 2 finds that effective tools stayed effective under four planned changes while monitoring signals raised false alarms; in an exploratory scenario, a tool refitted each period lost most of a risk type while its exceptions, fixed in number by the review budget, kept their usual spread across accounts. Study 3 finds that known-answer tests estimate detection within 0.05 on average, erring low for tools that rely on rarity, and falsify unfit tools cheaply, but speak only to cases like those seeded; remainder samples of affordable size rarely reveal a wholly missed condition. The rates describe one synthetic generator and one suite of tools; they show mechanisms, not their prevalence in practice. The paper proposes draft application material for ED-500 and ED-330. Not peer reviewed. The studies were specified in advance in a versioned design document with a dated change log; the document was not deposited with a third party before the studies ran. One scenario and two checks were added after results had been seen, and are labelled exploratory or post hoc throughout. The author draws on this study in responses to the AUASB and IAASB consultations on the proposed ISA 330, ISA 500 and ISA 520. How this was made: the author set the research question, designed the studies and made every analytical and editorial decision, and is responsible for the content. Code, analyses and drafts were prepared with Claude (Anthropic) under the author's direction, and each substantive claim was traced to its source in a claims ledger. Data and attribution: the experiments use the DIB General Ledger Data by Gronewald, Stephan, Rombach and Fettke (2026), Zenodo, doi:10.5281/zenodo.22014653, licensed CC BY 4.0, together with an adapted version of its generator notebook, used under CC BY 4.0. The changes are listed in the code, which is available from the author on request.

Zenodo (CERN European Organization for Nuclear Research)
Auditing, Earnings Management, Governance
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon