A survey on large language models unlearning: taxonomy, evaluations, and future directions

Abstract Following the introduction of data privacy regulations and “the right to be forgotten”, large language models (LLMs) unlearning has emerged as a promising data removal solution for compliance purposes, while also facilitating a diverse range of applications, including copyright protection, model detoxification and correction, and jailbreaking defence. In this survey, we present the taxonomy of existing LLMs unlearning algorithms, summarise unlearning evaluation methods including specialised benchmarks and threat models, and explore the applications of unlearning to provide a broad overview of the current state-of-the-art. We propose a novel problem formulation of LLMs unlearning with the additional unlearning objective: “robustness” to reflect the growing research interest in not only effectively and efficiently eliminating unwanted data, but also ensuring the process is performed safely and securely. To the best of our knowledge, we are the first to examine the robustness of unlearning algorithms as well as threat models for robustness evaluation, aspects that have not been assessed in past surveys. We also identify the limitations of the current approaches, including limited applicability to black-box models, vulnerability to adversarial attacks and knowledge leakage, and inefficiency, all of which require further improvement in future works. Furthermore, our survey highlights future directions for LLMs unlearning research, such as the development of comprehensive evaluation benchmarks, the movement towards robust unlearning and explainable AI for unlearning mechanisms, and addressing potential ethical dilemmas in unlearning governance.