Human-aligned AI for supply chain risk detection: validation and calibration of LLM-based early-warning assessments

Purpose This paper investigates how a large language model (LLM)-based early-warning system (EWS) for supply chain risk can be systematically aligned with human expert judgement. Building on prior work that introduced the overall system architecture, this study focuses on the detailed prompting logic underlying the automated analysis and on an empirical validation of the resulting AI-based risk assessments. The purpose is to examine how human evaluations can be used to validate and calibrate AI-generated classifications and numerical risk scores, thereby improving their accuracy, reliability and interpretability. In doing so, the paper contributes to a human-centric understanding of AI-supported decision-making in the context of Logistics 5.0. Design/methodology/approach The study draws on two datasets from a real-world validation conducted in January and March 2025, in which AI-generated risk indicators – including impact, likelihood, credibility, sentiment and risk category – were systematically compared with human expert assessments. Deviations between model and expert evaluations were analyzed using established performance metrics such as mean absolute error (MAE), classification accuracy and error distribution patterns. The methodological design follows a human-in-the-loop approach: expert assessments serve as an operational reference standard and are used to iteratively calibrate the prompting logic underlying the automated analysis. Quantitative evaluation is complemented by design-oriented insights derived from the iterative refinement of the system. Findings The findings demonstrate that a structured, iterative human-in-the-loop validation process successfully improves the quality of AI-based risk assessments. In the initial phase, limited contextual grounding led to conservative model behavior, particularly midpoint bias in numerical ratings such as severity, probability and temporal horizon. Prompt refinements systematically integrated supply-chain-relevant anchor examples, partly adapted to company-specific contexts. This enhanced the model's ability to interpret events in operational terms rather than abstract scales. Across the refined prompts, MAE decreased consistently, confirming that context-aware prompt calibration through expert feedback is an effective mechanism for aligning AI outputs with human judgement. Research limitations/implications The validation is based on two empirical evaluation rounds with a limited number of pilot companies, which constrains immediate generalization across industries and regions. Future research may extend the approach to larger datasets, additional sectors and alternative LLM variants. Nonetheless, the study provides a transferable methodological template for validating and calibrating AI-driven risk detection systems in logistics. More broadly, the findings underline the importance of embedding domain knowledge and expert judgement into AI-based classifications rather than relying solely on probabilistic model behavior, thereby supporting human-centric decision-making in line with Logistics 5.0 principles. Practical implications The results support logistics managers in assessing when and how AI-generated alerts can be trusted in operational risk monitoring. By documenting the prompting logic and workflow design, the study enables organizations to implement comparable EWSs using low-code environments. The findings show how structured human feedback reduces noise and misclassification, helping firms focus on truly critical disruptions, avoid unnecessary reactive measures and strengthen resilience and sustainability in supply chain operations. Social implications By aligning AI-generated assessments with human expertise, the proposed system reinforces human oversight rather than replacing it. This supports transparency, trust and responsible AI use in operational decision-making. In line with Logistics 5.0 principles, the approach reduces cognitive overload caused by information noise and enables more balanced, informed and socially responsible responses to supply chain disruptions. Originality/value This paper advances research on AI-based supply chain risk monitoring by providing a transparent and empirically grounded analysis of how LLMs can be calibrated through human expert judgement. Beyond presenting the prompting logic of an operational LLM-based EWS, the study demonstrates how structured expert feedback, context-specific anchor examples and iterative prompt refinement systematically improve model behavior across multiple risk dimensions. In contrast to prior conceptual, simulation-based or purely performance-driven studies, this work shows how and why human-in-the-loop calibration reduces systematic biases rather than merely reporting accuracy gains. By explicitly linking prompting logic, expert validation and operational decision logic, the paper contributes a transferable methodological approach and strengthens the practical and theoretical foundations of human-centric Logistics 5.0 risk management.