Contextual Reversals in Visual Explanations: Separating Intrinsic Disagreement from Ranking Error

Abstract A fixed regional ranking is often used to predict which image region will produce a larger model response when removed. Context can change this order, so disagreement may reflect both the prediction task and the selected ranking. This study audits these components by exhaustively evaluating a finite removal family on 100 fresh Imagenette images shared by MobileNetV3-Large and ResNet34. The primary cohort contains 68 originals correctly classified by both models. For each image, model and replacement operator, 1,820 size-four removal masks determine 43,680 pair/context comparisons. Across mean-fill and blur probes, the best fixed order incurs 17.26–23.95% mean disagreement, while additional ordering-constraint loss is only 0.0039–0.0110 percentage points. On evaluation masks disjoint from fitting masks, a 16-query singleton ranking incurs 2.44–5.21 additional points and outperforms signed Integrated Gradients and positive Grad-CAM in all four settings. Ninety-six-query fitted fixed rankings improve consistently; a context-conditioned predictor exceeds the fixed-order ceiling by 1.20–1.63 points. All primary blur-reference Integrated Gradients runs pass the numerical checks, whereas nine primary MobileNet mean-reference runs fail completeness and remain in the analysis. Disagreement persists under SLIC-derived regions on a matched 13-image subset. Independent integer programming certifies 960 optimization results. The findings support reporting a task-specific fixed-ranking ceiling alongside explainer agreement in finite removal-effect benchmarks.