Memory Pollution in Multi-Product Visual Anomaly Detection: Diagnosis and Mitigation

Memory-bank methods such as PatchCore are widely used in industrial quality control for visual anomaly detection since they require no training, are fast to deploy and achieve strong accuracy. However, they are memory-intensive. Furthermore, a single production line typically involves different products or cameras, so using a single anomaly detection method with a shared nearest-neighbor memory bank is attractive since it simplifies deployment and makes new products easy to add. Nevertheless, embeddings from different products/cameras can interfere during retrieval, causing what we call “memory pollution”. In this work, we study this effect through a new diagnostic framework, which involves: (1) a new metric, the wrong-neighbor rate (WNR), which measures how often a query’s nearest neighbor belongs to a different product; (2) an empirically validated phenomenon, “oracle inversion”, where querying only the product’s own data can underperform the shared bank under a fixed memory budget; (3) a first-order analytical model of the WNR, which predicts how pollution grows with product count and memory budget; and (4) a minimal training-free router that removes the effect of memory pollution. Our results show that our system performs robustly across different datasets and backbones, with up to 25× memory reduction, which makes our framework attractive for industrial applications.