Detecting Data Poisoning Attacks in Image Datasets Using a Vision-Language Hybrid Pipeline

Image classification systems play a crucial role in real-world applications. However, these systems are vulnerable to data poisoning attacks, where attackers manipulate training data to degrade model accuracy. Existing defenses often struggle to detect manipulations. In this work, we build on a prior model, DynaDetect2.0, which uses Convolutional Neural Network (CNN) based feature extraction and statistical outlier detection. We extend this framework by integrating a Vision Language Model (VLM) to support semantic reasoning during detection. The VLM operates at two stages: it first pre-screens image-label pairs for semantic alignment and later provides natural-language explanations for samples flagged by statistical methods. This hybrid approach combines statistical rigor with semantic interpretability. We evaluate the proposed method using the Imagenette-160 and CIFAR-100 datasets with both clean and poisoned data. Results show that our VLM-assisted detection reduces false positives and improves the identification of mislabeled or anomalous data compared to statistical models alone.