Can Occupational Safety Practitioners Trust AI-Ranked Risk Factors? A Synthetic Study of Predictive Performance and Explanation Fidelity

Artificial intelligence tools are increasingly used in occupational safety to rank factors associated with incidents and to support decisions on training, supervision, inspection, and preventive resource allocation. We evaluated whether predictive performance and explanation fidelity remain aligned when events are rare, and predictors are correlated. A fully synthetic occupational system was generated with Safety Data Sheet competence, six Bloom’s Taxonomy dimensions, behavioral and organizational variables, and a simulated incident outcome. Random forests and ridge models were assessed for nested cross-validation discrimination and calibration; random forests were additionally evaluated using ablation, held-out marginal and conditional permutation feature importance, and matched oracle importance derived from the known data-generating mechanism. A 29-condition Monte Carlo experiment (58,000 random-forest fits) examined sample size, prevalence, effect magnitude, and predictor architecture. Under the baseline condition, mean AUC was 0.689, whereas oracle–model rank agreement was weak (Spearman ρ=0.048) and oracle top-1 recovery was 20.7%. A high-signal implementation check produced ρ=0.964, with 83.4% top-1 and 98.5% top-2 recovery, while a more intensive rare-event pipeline left top-1 recovery essentially unchanged at 20.3%. Conditional permutation importance, benchmarked against a matched conditional oracle, reduced spurious first-ranking of true-null controls (26.9% to 22.9%, p<0.001) but did not improve recovery of the oracle top variable (20.4% versus 20.7%). AI-derived factor rankings should therefore be validated separately from predictive performance before they are used to set preventive priorities within conventional occupational risk-assessment processes. The evidence presented here concerns random forests with marginal and conditional permutation importance under fully synthetic rare-event conditions, and does not generalize to AI explanation methods as a class.

Authors

Institutions

Publication Details

Journal
Safety
Published
2026-11-12
DOI
https://doi.org/10.3390/safety12060148
Primary Topic
Occupational Health and Safety Research
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Can Occupational Safety Practitioners Trust AI-Ranked Risk Factors? A Synthetic Study of Predictive Performance and Explanation Fidelity

Ana Delgado-García, Raúl Aguilar-Elena, Alejandro Guillén‐Riquelme, Juan José Agún González et al.
Safety
Occupational Health and Safety Research
article

Can Occupational Safety Practitioners Trust AI-Ranked Risk Factors? A Synthetic Study of Predictive Performance and Explanation Fidelity

Ana Delgado-García, Raúl Aguilar-Elena, Alejandro Guillén‐Riquelme, Juan José Agún González, Rubén Tino-Ramos
article en

Abstract

Artificial intelligence tools are increasingly used in occupational safety to rank factors associated with incidents and to support decisions on training, supervision, inspection, and preventive resource allocation. We evaluated whether predictive performance and explanation fidelity remain aligned when events are rare, and predictors are correlated. A fully synthetic occupational system was generated with Safety Data Sheet competence, six Bloom’s Taxonomy dimensions, behavioral and organizational variables, and a simulated incident outcome. Random forests and ridge models were assessed for nested cross-validation discrimination and calibration; random forests were additionally evaluated using ablation, held-out marginal and conditional permutation feature importance, and matched oracle importance derived from the known data-generating mechanism. A 29-condition Monte Carlo experiment (58,000 random-forest fits) examined sample size, prevalence, effect magnitude, and predictor architecture. Under the baseline condition, mean AUC was 0.689, whereas oracle–model rank agreement was weak (Spearman ρ=0.048) and oracle top-1 recovery was 20.7%. A high-signal implementation check produced ρ=0.964, with 83.4% top-1 and 98.5% top-2 recovery, while a more intensive rare-event pipeline left top-1 recovery essentially unchanged at 20.3%. Conditional permutation importance, benchmarked against a matched conditional oracle, reduced spurious first-ranking of true-null controls (26.9% to 22.9%, p<0.001) but did not improve recovery of the oracle top variable (20.4% versus 20.7%). AI-derived factor rankings should therefore be validated separately from predictive performance before they are used to set preventive priorities within conventional occupational risk-assessment processes. The evidence presented here concerns random forests with marginal and conditional permutation importance under fully synthetic rare-event conditions, and does not generalize to AI explanation methods as a class.

SafetyVol. 12(6)
Universidad de Salamanca (ES), Valencian International University (ES)
Good health and well-being
Occupational Health and Safety Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon