Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches

ABSTRACT Delphi studies in disaster medicine lack consensus on expert agreement metrics. This study examined various metrics using a Delphi study on chemical, biological, radiological, and nuclear (CBRN) preparedness in the Middle East and North Africa region. Forty international disaster medicine experts evaluated 133 items across ten CBRN Preparedness Assessment Tool themes using a 5‐point Likert scale. Agreement was measured using Kendall's W, Intraclass Correlation Coefficient, and Cohen's Kappa. Statistical and machine learning techniques compared metric performance. The overall agreement mean score was 4.91 ± 0.71, with 89.21% average agreement. Kappa emerged as the most sensitive metric in statistical and machine learning analyses, with a feature importance score of 168.32. The Kappa coefficient showed variations across CBRN PAT themes, including medical protocols, logistics, and infrastructure. The integrated statistical and machine learning approach provides a promising method for understanding expert consensus in disaster preparedness, with potential for future refinement by incorporating additional contextual factors.

Authors

Institutions

Publication Details

Journal
Journal of Contingencies and Crisis Management
Published
2025-04-06
DOI
https://doi.org/10.1111/1468-5973.70044
Citations
1
Primary Topic
Risk and Safety Analysis
Type
article
Field-Weighted Citation Impact
1.40
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches

Mariana Charbel Helou, H. Farhat, Gregory R. Ciottone, James Laughton et al.
1 citations
Journal of Contingencies and Crisis Management
Risk and Safety Analysis
1.40
article

Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches

Mariana Charbel Helou, H. Farhat, Gregory R. Ciottone, James Laughton, Alan M Batt, Roberto Mugavero, Nidaa Bajow, Luc Mortelmans, Arezoo Dehghani, Guillaume Alinier, Walid Abougalala, Foroogh Rezaei, Mohamed Ben Dhiab, Heejun Shin, Carolyn Dumbeck
article en
1 citations

Abstract

ABSTRACT Delphi studies in disaster medicine lack consensus on expert agreement metrics. This study examined various metrics using a Delphi study on chemical, biological, radiological, and nuclear (CBRN) preparedness in the Middle East and North Africa region. Forty international disaster medicine experts evaluated 133 items across ten CBRN Preparedness Assessment Tool themes using a 5‐point Likert scale. Agreement was measured using Kendall's W, Intraclass Correlation Coefficient, and Cohen's Kappa. Statistical and machine learning techniques compared metric performance. The overall agreement mean score was 4.91 ± 0.71, with 89.21% average agreement. Kappa emerged as the most sensitive metric in statistical and machine learning analyses, with a feature importance score of 168.32. The Kappa coefficient showed variations across CBRN PAT themes, including medical protocols, logistics, and infrastructure. The integrated statistical and machine learning approach provides a promising method for understanding expert consensus in disaster preparedness, with potential for future refinement by incorporating additional contextual factors.

Journal of Contingencies and Crisis ManagementVol. 33(2)
Isfahan University of Medical Sciences (IR), University of Rome Tor Vergata (IT), Harvard University (US), University of Hertfordshire (GB), Queen's University (CA), Soonchunhyang University (KR), University Medical Center Rizk Hospital (LB), Harvard University Press (US), Northumbria University (GB), University of the Republic of San Marino (SM), Security Forces Hospital (SA), Weill Cornell Medical College in Qatar (QA), Hamad Medical Corporation (QA), Lebanese American University (LB), Monash University (AU), Shahid Beheshti University of Medical Sciences (IR), Bucheon University (KR), University of Sousse (TN), KU Leuven (BE)
Openalex Percentile: Top 17%
Risk and Safety Analysis
1.40
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.