Slow Drift Temporal Poisoning attacks and vision language model guided defense for BEV perception in autonomous vehicles

World-model BEV perception maintains a persistent temporal latent memory that can be gradually corrupted by perturbations too small to trigger existing per-frame anomaly checks. We formalize this as Slow-Drift Temporal Poisoning (SDTP): frame-wise perturbations bounded by a small digital L∞ budget (ε ≤ 2/255) - a standard proxy for low perturbation magnitude, not a validated measure of human-perceptual imperceptibility or physical-world realizability, jointly optimized across a temporal window via a Backward Temporal Gradient procedure, that reach an 84.3% Temporal Drift Success Rate by frame 32 on BEVFormer-T while evading temporal-consistency defenses calibrated on single-step thresholds. We propose SemantiGuard, a cross-modal defense that cross-checks BEV detections against a frozen vision-language model’s scene description and triggers latent-memory rollback on divergence, reducing the Temporal Drift Success Rate (TDSR) from 91.2% to 13.8% under the primary evaluation configuration (SDTP-L: ε = 4/255, T = 32, BEVFormer-T, Table 6), with a lower 11.2% TDSR observed under the smaller SDTP-M ablation configuration (ε = 2/255, T = 16, Table 7). We also introduce TemporalAdvBEV, a benchmark extending RoboBEV with multi-frame adversarial configurations and drift-specific metrics. These results indicate that temporal memory is an attack surface that BEV security evaluation has not yet accounted for, and that cross-modal semantic verification is a promising, if not yet field-validated, direction for defending it. The paper also introduces TemporalAdvBEV, a benchmark extending RoboBEV with multi-frame adversarial configurations and drift-specific metrics, designed to support evaluation on both nuScenes and Waymo Open, with nuScenes results reported in this manuscript.