How Scoring Responses Change Attribution Evaluation in a Diffusion Policy

Feature attributions rank inputs associated with a model's prediction. Their evaluation depends on how a scalar response scores predictions after features are removed or restored. For robot policies, we separate the attribution target from the scoring response. We compare squared distance (Q) and stabilized distance (N) from a fixed reference action. An identity shows that N cannot increase insertion or deletion area under the curve (AUC) when no intervention overshoots, that is, moves the prediction farther from the reference, in squared distance, than replacing every feature with its baseline. Strict decreases make an unchanged ranking look worse by insertion and better by deletion; absolute thresholds can give opposite verdicts. We rescore saved curves for PickCube, a simulated pick-and-place task, from a model recorded as a pretrained Robotics Diffusion Transformer. Q-ranked and N-ranked cohorts ranked features using Q and N, respectively; each contains 750 policy calls from 30 episodes with no successes. Across eight combinations of cohort, modality and direction, median Q-minus-N differences are 0.104 to 0.159, with 75.6% to 94.3% positive. Overshoots occur in 16.0% to 74.0% of calls. Extreme calls, mainly each episode's first prediction, make vision-insertion means strongly negative. An example shows that target choice can also reverse rankings. Separate numerical diagnostics show that integration error can change evaluation results or leave them unchanged. A smooth counterexample shows that agreeing uniform integration grids can give incorrect coordinates. We recommend specifying targets and responses separately, comparing rankings under matched controls, reporting full distributions, and checking numerical sensitivity.