MAGNET: Counterfactual samples synthesizing for mitigating hallucination in large language models

Hallucinations are widely recognized as a significant drawback of large language models. Several attempts have been made to reduce the intensity of hallucinations. Among the various attempts, our research has been directed towards mitigating hallucinations caused by the co-occurrence statistics of pre-training corpora. We introduce Model-AGNostic countErfacTual synthesis and adaptive fine-tuning framework (MAGNET), a fine-tuning method that can mitigate the bias of co-occurrence statistics on large language models pre-training data when generating sentences. Our pipeline generates the counterfactual sample sentences and subject and object information for the counterfactual sample from the language model, and filters them to make sure they contain these three pieces of information before using them as fine-tuning data. Next, it utilizes both the generated counterfactual sample and the original sentence used to generate it as a training dataset. When our method is applied to GPT-Neo 2.7B model, it shows a 12% improvement in the Factual Knowledge Probing experiment, and there is a correlation analysis that can mitigate the bias on the pre-training data. In the TruthfulQA experiment, when fine-tuning the GPT-Neo 125M model on the LAMA-TREx dataset, applying our method showed 2.27% better performance than not applying it.