Exposing Patient-Level Data Leakage in Respiratory Sound Classification: A Leakage-Free Benchmark Leveraging Transformer-Based TabPFN-3 and Knowledge Distillation

Background/Objectives: Many studies on automated respiratory sound classification report near-perfect accuracy. In most of these studies, respiratory cycles are split randomly into training and test sets, so cycles of the same patient can be found in both sets. This patient-level data leakage makes the reported results higher than the real performance. In this study, we show this problem on the ICBHI 2017 dataset and build a leakage-free benchmark. Methods: We compare TabPFN-3, a pretrained tabular foundation model that predicts by in-context learning without task-specific gradient updates, with eight standard machine learning classifiers in a three-stage framework. Results: When the cycle-level split was replaced by a patient-level split (a stratified five-fold split repeated three times, 15 test folds, with every class present in every test fold), the accuracy of TabPFN-3 decreased from 0.9943 to 0.8722, the macro-F1 from 0.9807 to 0.5010 and the Matthews Correlation Coefficient (MCC) from 0.9805 to 0.5788. Nadeau–Bengio-corrected t-tests with Holm adjustment over the 15 folds showed that TabPFN-3 had a significantly higher MCC than all eight baselines. SHAP analysis showed that the first MFCC coefficient, the spectral bandwidth and the RMS energy contributed most to the predictions. We also distilled TabPFN-3 into small student models. The best student, a small transformer (276.3 KB), reached an MCC of 0.5140; its MCC difference from the teacher was not significant after Holm correction, but an equivalence test did not show equivalence and its macro-F1 was significantly lower. A shallow distilled decision tree gave readable rules, mainly for COPD and healthy recordings. Conclusions: Performance on several minority diseases remains low, so the models are not ready for clinical use, and the results should be seen as a realistic baseline for future work.

Authors

Institutions

Publication Details

Journal
Diagnostics
Published
2026-11-17
DOI
https://doi.org/10.3390/diagnostics16223834
Primary Topic
Phonocardiography and Auscultation Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Exposing Patient-Level Data Leakage in Respiratory Sound Classification: A Leakage-Free Benchmark Leveraging Transformer-Based TabPFN-3 and Knowledge Distillation

Damla Gurkan Kuntalp, Fevzi Yasin Kababulut, Mehmet Kuntalp, Okan Düzyel
2 links
Diagnostics
Phonocardiography and Auscultation Techniques
article

Exposing Patient-Level Data Leakage in Respiratory Sound Classification: A Leakage-Free Benchmark Leveraging Transformer-Based TabPFN-3 and Knowledge Distillation

Damla Gurkan Kuntalp, Fevzi Yasin Kababulut, Mehmet Kuntalp, Okan Düzyel
article en

Abstract

Background/Objectives: Many studies on automated respiratory sound classification report near-perfect accuracy. In most of these studies, respiratory cycles are split randomly into training and test sets, so cycles of the same patient can be found in both sets. This patient-level data leakage makes the reported results higher than the real performance. In this study, we show this problem on the ICBHI 2017 dataset and build a leakage-free benchmark. Methods: We compare TabPFN-3, a pretrained tabular foundation model that predicts by in-context learning without task-specific gradient updates, with eight standard machine learning classifiers in a three-stage framework. Results: When the cycle-level split was replaced by a patient-level split (a stratified five-fold split repeated three times, 15 test folds, with every class present in every test fold), the accuracy of TabPFN-3 decreased from 0.9943 to 0.8722, the macro-F1 from 0.9807 to 0.5010 and the Matthews Correlation Coefficient (MCC) from 0.9805 to 0.5788. Nadeau–Bengio-corrected t-tests with Holm adjustment over the 15 folds showed that TabPFN-3 had a significantly higher MCC than all eight baselines. SHAP analysis showed that the first MFCC coefficient, the spectral bandwidth and the RMS energy contributed most to the predictions. We also distilled TabPFN-3 into small student models. The best student, a small transformer (276.3 KB), reached an MCC of 0.5140; its MCC difference from the teacher was not significant after Holm correction, but an equivalence test did not show equivalence and its macro-F1 was significantly lower. A shallow distilled decision tree gave readable rules, mainly for COPD and healthy recordings. Conclusions: Performance on several minority diseases remains low, so the models are not ready for clinical use, and the results should be seen as a realistic baseline for future work.

DiagnosticsVol. 16(22)
Izmir Institute of Technology (TR), Dokuz Eylül University (TR)
Good health and well-being
Phonocardiography and Auscultation Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon