Temporal and sleep stage‐dependent agreement in manual scoring of respiratory events

Obstructive sleep apnea diagnosis is based on the manual scoring of respiratory events. The agreement in the manual scoring of the respiratory events lacks an in-depth investigation as most of the previous studies reported only the apnea-hypopnea index or overall agreement, and not temporal, second-by-second or event subtype agreement. We hypothesized the temporal and subtype agreement to be low because the event duration or subtypes are not generally considered in current clinical practice. The data comprised 50 polysomnography recordings scored by 10 experts. The respiratory event agreement between the scorers was calculated using kappa statistics in a second-by-second manner. Obstructive sleep apnea severity categories (no obstructive sleep apnea/mild/moderate/severe) were compared between scorers. The Fleiss' kappa value for binary (event/no event) respiratory event scorings was 0.32. When calculated separately within N1, N2, N3 and R, the Fleiss' kappa values were 0.12, 0.23, 0.22 and 0.23, respectively. Binary analysis conducted separately for the event subtypes showed the highest Fleiss' kappa for hypopneas to be 0.26. In 34% of the participants, the obstructive sleep apnea severity category was the same regardless of the scorer, whereas in the rest of the participants the category changed depending on the scorer. Our findings indicate that the agreement of manual scoring of respiratory events depends on the event type and sleep stage. The manual scoring has discrepancies, and these differences affect the obstructive sleep apnea diagnosis. This is an alarming finding, as ultimately these differences in the scorings affect treatment decisions.

Authors

Institutions

Publication Details

Journal
Journal of Sleep Research
Published
2024-11-04
DOI
https://doi.org/10.1111/jsr.14391
Citations
2
Primary Topic
Obstructive Sleep Apnea Research
Type
article
Field-Weighted Citation Impact
0.65

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Temporal and sleep stage‐dependent agreement in manual scoring of respiratory events

Samu Kainulainen, Rajdeep Kumar Nath, Tarja Saaresranta, Jan Anders Hedner et al.
2 citations
Journal of Sleep Research
Obstructive Sleep Apnea Research
0.65
article

Temporal and sleep stage‐dependent agreement in manual scoring of respiratory events

Samu Kainulainen, Rajdeep Kumar Nath, Tarja Saaresranta, Jan Anders Hedner, Sami Nikkonen, Sigríður Sigurðardóttir, Ulla Anttalainen, Henna Pitkänen, Ludger Grote, Thomas Penzel, Richard Staats, Juha Töyräs, Francesco Fanfulla, Minna Pitkänen, Henri Korkalainen, Timo Leppänen, Erna Sif Arnardóttir, Kristín A. Ólafsdóttir
article en
2 citations

Abstract

Obstructive sleep apnea diagnosis is based on the manual scoring of respiratory events. The agreement in the manual scoring of the respiratory events lacks an in-depth investigation as most of the previous studies reported only the apnea-hypopnea index or overall agreement, and not temporal, second-by-second or event subtype agreement. We hypothesized the temporal and subtype agreement to be low because the event duration or subtypes are not generally considered in current clinical practice. The data comprised 50 polysomnography recordings scored by 10 experts. The respiratory event agreement between the scorers was calculated using kappa statistics in a second-by-second manner. Obstructive sleep apnea severity categories (no obstructive sleep apnea/mild/moderate/severe) were compared between scorers. The Fleiss' kappa value for binary (event/no event) respiratory event scorings was 0.32. When calculated separately within N1, N2, N3 and R, the Fleiss' kappa values were 0.12, 0.23, 0.22 and 0.23, respectively. Binary analysis conducted separately for the event subtypes showed the highest Fleiss' kappa for hypopneas to be 0.26. In 34% of the participants, the obstructive sleep apnea severity category was the same regardless of the scorer, whereas in the rest of the participants the category changed depending on the scorer. Our findings indicate that the agreement of manual scoring of respiratory events depends on the event type and sleep stage. The manual scoring has discrepancies, and these differences affect the obstructive sleep apnea diagnosis. This is an alarming finding, as ultimately these differences in the scorings affect treatment decisions.

Journal of Sleep ResearchVol. 34(3)
Reykjavík University (IS), University of Lisbon (PT), University of Turku (FI), The University of Queensland (AU), University of Eastern Finland (FI), Turku University Hospital (FI), Istituti Clinici Scientifici Maugeri (IT), Kuopio University Hospital (FI), Charité - Universitätsmedizin Berlin (DE), VTT Technical Research Centre of Finland (FI), University of Gothenburg (SE)
European Commission, Suomen Kulttuurirahasto, Hjärt-Lungfonden, Kuopion Yliopistollinen Sairaala, Magnus Ehrnroothin Säätiö, Sigrid Juséliuksen Säätiö, Horizon 2020 Framework Programme
Good health and well-being
Openalex Percentile: Top 28%
Obstructive Sleep Apnea Research
0.65
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.