Advances in the Identification and Agreement About Meta-Therapy for Voice Disorders: A Methodological Paper

BACKGROUND: Meta-therapy (MT) is a powerful dialogue-based element of voice therapy that scaffolds patients' cognitive models of treatment. MT dialogues have not historically been taught in an explicit manner, and substantial variability in identifying MT exists within and across clinicians. This low reliability is problematic for empirical research and educational transfer. Harnessing contemporary natural language processing technologies to stress test the concept of MT and its clinical instances may enhance our theoretical and empirical grasp of the construct. METHOD: To capture crucial therapeutic component parts and MT, 10,443 clinician utterances stemming from conversation training therapy sessions delivered by six expert voice-specialized speech-language pathologists were transcribed and analyzed with a refined annotation framework. Two independent raters annotated each session and reconciled disagreements through adjudication, and the resulting consensus was compared with an expert gold standard. Time distribution was analyzed. Linguistic profiles were derived from bigram frequencies in utterances reaching expert-annotator consensus. Reliability and ambiguity in identification were assessed with multilabel confusion matrices, percentage agreement, Cohen's kappa, and Gwet's agreement coefficient (AC1). RESULTS: Gwet's AC1 indicated substantial-to-almost-perfect agreement for MT in most sessions, outperforming κ and mitigating base-rate artifacts (AC1: up to .98). Mean stand-alone MT duration ranged from 0.32 to 3.88 min per session. Distinctive MT bigrams contrasted with motor practice or psychosocial collocations that typified direct and counseling content, although lexical overlap produced systematic confusions with MT blended with direct and education/indirect labels. CONCLUSIONS: The revised annotation model markedly improved the reproducibility of MT identification and revealed its linguistic signature. The model confirmed MT's role as a cross-cutting discourse that integrates with other therapeutic modalities. These advances provide an empirical foundation for machine learning classifiers, formal curriculum content, and further investigation into MT's contribution to treatment efficacy. SUPPLEMENTAL MATERIAL: https://doi.org/10.23641/asha.32159277.