The Application of Membership Inference in Privacy Auditing of Large Language Models Based on Fine-Tuning Method

Thanks to the large datasets collected from the internet, large language models (LLMs) have made significant progress across various fields, with widespread applications in digital employees, intelligent customer service, and smart decision-making. However, during the training process, models may memorize sensitive private data, posing risks to data security and privacy protection. Member Inference Attacks (MIA) aim to determine whether a specific record was used in the training of the target model. This approach is crucial for detecting privacy risks in models and has seen substantial progress in both traditional and fine-tuned models. However, this remains an unresolved issue for large language models (LLMs). Recent studies indicate that under proper experimental setups, the AUC-ROC of existing methods approaches 0.5 (random guessing). Subsequently, some research has shifted MIA to dataset inference, achieving promising results in this area. Our study shows that when fine-tuning a target model with aggregated samples, it is possible to distinguish whether these samples were used in training the target model by observing the initial loss returned by the model. Based on this observation, we propose a fine-tuning-based member inference method for large language models. To validate the effectiveness of our method, we conducted extensive experiments on three mainstream models and seven datasets. The results indicate that member inference attacks with aggregated samples can achieve acceptable outcomes.