Participant Overlap and Apparent Accuracy in Behavioral Depression Classification

Abstract Background Behavioral machine-learning studies can include several observations from each participant. When observations from the same person appear in both training and evaluation data, a classifier may recognize participant-specific characteristics instead of learning relationships that generalize to new people. Objective To quantify how participant overlap affects apparent classification accuracy under controlled conditions with and without a class-related signal. Methods Six simulation scenarios combined three levels of participant-specific variation with two levels of class-related signal. Each scenario contained 100 Monte Carlo replicates, each comprising 60 simulated participants, 10 observations per participant, and 10 features. A one-nearest-neighbor classifier was evaluated using five-fold record-wise and participant-wise cross-validation. Both approaches used 480 training observations and 120 evaluation observations per fold. Outcomes were pooled out-of-fold accuracy, the paired accuracy difference between evaluation approaches, and the frequency with which the record-wise classifier selected a neighbor from the same participant. Results Without class-related signal, increasing participant-specific variation raised mean record-wise accuracy from 49.88% to 94.81%. Corresponding participant-wise accuracy ranged from 49.90% to 51.27%. At the strongest participant-specific variation, the mean accuracy difference was 43.54 percentage points, with a Monte Carlo standard error of 0.41 percentage points. In that scenario, 89.81% of selected neighbors belonged to the same participant as the evaluated observation. Conclusions Under the simulated conditions, high record-wise accuracy occurred without a generalizable class-related signal. Evaluation intended to estimate performance in previously unseen people should preserve participant separation. These findings demonstrate an evaluation mechanism and do not establish depression-detection performance.