A General Framework for Defending against Query-based Black-box Attacks on Neural Network Models

As Machine Learning as a Service (MLaaS) becomes increasingly prevalent, query-based black-box attacks pose a significant threat to machine learning systems. These attacks manipulate input queries to generate adversarial examples, resulting in misclassification or undesired outputs. To address these threats, researchers have developed various query moderation techniques. However, recent studies have revealed that these techniques are vulnerable to sophisticated adaptive attacks. In this paper, we propose a general framework, Query Provenance Analysis (QPA), to defend against query-based black-box attacks both robustly (against non-adaptive and adaptive attacks) and efficiently (in real-time). Our key insight is that, instead of focusing on individual queries, utilizing features from the query sequence (termed query provenance ) can distinguish malicious queries from benign queries more effectively. We construct a query provenance graph to capture the relationship between a new query and prior historical queries, and then design efficient algorithms to detect malicious queries based on the query provenance graphs. We evaluate QPA on image classification and Natural Language Processing (NLP) tasks. QPA significantly reduces the Attack Success Rate (ASR) compared to baselines, achieving up to 20 \(\times\) lower ASR for image classification and 10 \(\times\) lower ASR for NLP, while also improving throughput and latency across all tasks.