SENTINEL-X: Evaluating Context-Aware SOC Alert Decision Reliability Under Cross-Environment and Temporal Distribution Shifts

The Security Operation Centers (SOCs) are using artificial intelligence (AI) and machine learning (ML) more and more for detecting and prioritizing security alerts. While these methods could prove to be effective, there are still issues regarding the shifting distributions and changing realities in which the solutions still need to be tested. This poses a challenge in terms of generalization and long-term effectiveness of AI-driven security operations centers in Security Operation Center (SOC). To deal with this, we introduce SENTINEL-X, an experimental framework designed to evaluate the robustness of context-aware Security Operation Center (SOC) decision-making under distribution shift. The scope of this research encompasses two primary modalities of distribution shift: environmental discrepancies and time-based data evolution. Our framework provides a holistic evaluation of the SOC decision-making process, spanning threat detection, context driven prioritization, uncertainty estimation and the final determination to either automate actions or escalate them to a security analyst. To ensure operational realism, SENTINEL-X bypasses basic classification metrics to directly measure alert ranking, decision trustworthiness, automation coverage, erroneous actions and analyst workload. We put different setups head-to-head to find out if context builds a more resilient AI and if a smart “defer” button can stop automated mistakes before they happen. The objective of this research is to establish a comprehensive, unified evaluation of AI-driven SOC workflows under shifting conditions, thereby exposing critical limitations that static or single-environment test beds fail to reveal.