Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models

This paper explores Machine Unlearning (MU), an emerging field that is gaining increased attention due to concerns about neural models unintentionally remembering personal or sensitive information. We present SeUL, a novel method that enables selective and fine-grained unlearning for language models. Unlike previous work that employs a fully reversed training objective in unlearning, SeUL minimizes the negative impact on the capability of language models, particularly in terms of generation. Furthermore, we introduce two innovative evaluation metrics, sensitive extraction likelihood (S-EL) and sensitive memorization accuracy (S-MA), specifically designed to assess the effectiveness of forgetting sensitive information. In support of the unlearning framework, we propose efficient automatic online and offline sensitive span annotation methods. The online selection method, based on language probability scores, ensures computational efficiency, while the offline annotation involves a two-stage LLM-based process for robust verification. In summary, this paper contributes a novel selective unlearning method (SeUL), introduces specialized evaluation metrics (S-EL and S-MA) for assessing sensitive information forgetting, and proposes automatic online and offline sensitive span annotation methods to support the overall unlearning framework and evaluation.

Authors

• Lingzhi Wang Harbin Institute of Technology, Shenzhen, China
• Xingshan Zeng Huawei Noah's Ark Lab, Hong Kong, China
• Jinsong Guo Unlimidata Ltd, United Kingdom
• Kam-Fai Wong Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong MoE Key Lab. of High Confidence Software Technologies(Hong Kong), China
• Georg Gottlob

Keywords

No keywords are indexed for this paper.