Inter-rater Reliability Validation Via The Military Training Mental Toughness Inventory In Firefighters: Follow-up Psychometric Analysis

INTRO: Mental toughness (MT) is a critical psychological attribute, often assessed using cross-sectional self-assessment methods. Research links MT to superior performance, particularly within high-stress domains like tactical environments (e.g., firefighting). The Military Training Mental Toughness Inventory (MTMTI) is an established tool for measuring MT from a tactical perspective. Despite the relevance of MT in high-pressure professions, research in this context is limited, and specific instruments for assessing firefighting-related MT are absent. Preliminary work on examining the inter-rater reliability of MTMTI on firefighters concluded that there was no agreement on MT peer-rating by using the MTMTI. Since still there is no specialized instrument tailored to evaluate MT in firefighting, continuation of data was deemed justified. PURPOSE:To validate the inter-rater reliability of MTMTI in firefighters in a two-point longitudinal design. METHODS: Sixty-two male firefighters from two southern Florida fire departments participated in this study, completing an MTMTI survey over two days. The survey was administered by a colleague and an officer who conducted peer-ratings. The MTMTI comprises six items, each rated on a 7-point Likert scale, evaluating an individual's ability to maintain performance under stressful conditions. Statistical measures, including Cohen’s kappa (κ), Cronbach’s alpha (α), McDonald’s omega (ω), and the Interclass correlation coefficient (ICC), were employed for inter-rater reliability assessment. The analysis was performed using R statistical packages within Jamovi version 2.4.8 (p < .05). RESULTS: The inter-rater reliability, indicated by the Kappa coefficient, was minimal on both days, with low agreement between raters. Internal consistency measures (Alpha and Omega) were also unacceptable on both days. Inter-rater correlations were weak CONCLUSIONS: The study's findings do not demonstrate concurrence between the two raters, reinforcing previous data. Consequently, the inferences drawn regarding the MTMTI scores assigned by these two distinct raters also lack accord. Factors contributing to this lack of agreement should be explored. Future research should aim to refine measurement tools and explore the multifaceted nature of MT within firefighting.