Socio‐Technical Risk‐Informed AI‐Driven Automation Trustworthiness Evaluation (ST‐RATE): Part 1‐Conceptualization

ABSTRACT The growing integration of artificial intelligence (AI)‐driven technologies into nuclear power plant (NPP) operations and maintenance holds significant potential to enhance safety, operational efficiency, and reliability. However, deploying such AI‐driven automation technologies in high‐consequence settings requires a rigorous, traceable, and risk‐informed evaluation of their trustworthiness. Although researchers across various domains have proposed definitions and metrics for AI‐driven automation trustworthiness, no agreed‐upon formal or quantitative method exists that is suitable for the highly regulated nuclear sector. To address this gap, this paper, the first in a two‐part series, presents three contributions: (I) a cross‐disciplinary literature review that categorizes existing definitions and evaluation methodologies for AI‐driven automation trustworthiness and identifies their inherent limitations with respect to applicability in the nuclear domain; (II) develops a new definition of AI‐driven automation trustworthiness as “the degree of confidence that the AI‐driven automation system will function as expected across its conditions of use” to address the limitations identified in Contribution I; and (III) building on the new definition, conceptualizes the Socio‐Technical Risk‐informed AI‐driven Automation Trustworthiness Evaluation (ST‐RATE) methodology, grounded in three foundational pillars: (a) risk‐informed performance expectations, (b) socio‐technical context across operational conditions, and (c) uncertainty‐based trustworthiness evaluation. While developed primarily for the nuclear domain, the definition and ST‐RATE methodology has the potential to be adapted for other safety‐critical domains with appropriate modifications. The ST‐RATE methodological framework is operationalized and demonstrated through an AI‐driven automated firewatch case study in a companion paper (Companion paper Part 2, 2025). ST‐RATE is the first‐of‐its‐kind methodology that enables risk‐informed and uncertainty‐based quantification of trustworthiness despite limited empirical data and without over‐reliance on subjective expert judgment, addressing the key limitations of existing approaches identified in the literature review.