Naturalistic Social Dyads Assessment In Free Play: A Multi-view Framework For Recognizing Co-located Eye Contact Between Children and Their Assessors

Abstract Mutual eye contact is a clinically relevant behavioral marker in social engagement assessments. Existing clinical tools often rely on manual observation, which may introduce subjective biases. No validated video-derived algorithm currently exists for objectively recognizing clinically defined eye contact behaviors in multi-individual free play settings. This study introduced a novel multi-view deep learning framework for automated recognition of clinically defined eye contact behaviors between children and assessors during free play. We investigated whether multi-view video recordings and spatial-temporal behavioral patterns improved recognition performance. The study included video recordings from 103 children diagnosed with autism (mean age = 6.96 years) and 33 typically developing children (mean age = 7.63 years). Mutual eye contact behaviors between children and assessors were analyzed during five-minute free play sessions, with caregivers present in the assessment room. The framework was trained using sparsely sampled spatial- and temporal-domain images from multiple fixed cameras capturing complementary views of the assessment environment. The fused single-view model achieved F1 scores of 0.85 [0.83, 0.88] for eye contact behavior and 0.90 [0.89, 0.91] for non-eye-contact behavior. Multi-view fusion improved performance to 0.92 [0.89, 0.95] and 0.94 [0.93, 0.96], respectively. These findings suggest that multi-view video analysis can objectively identify clinically defined eye contact behaviors among multiple individuals during clinical free play assessments. The framework may support scalable behavioral coding and quantitative assessment of clinically relevant social behaviors consistent with established criteria.