Erect: Evidence Refinement Enhanced Complex Claim Verification With Large Language Models

Automatic claim verification aims to predict the veracity of claims automatically. Large Language Models (LLMs) have been leveraged to check the veracity of claims and achieved great progress, due to their powerful logical reasoning ability and rich knowledge reserve. However, since the LLMs have factual knowledge boundary problem that comes from outdated and incomplete pre-training data, they may experience hallucinations when verifying emerging claims. Evidence retrieval is helpful in improving the performance of LLM-based fact verification. However, the quality of retrieved evidence directly affects the verification effectiveness of LLMs. In this paper, we present ERECT, an Evidence Refinement Enhanced complex Claim verificaTion model, which refines more accurate evidence with the help of LLMs to further assist LLMs in verifying complex claims more accurately. Specifically, ERECT first decomposes complex claim verification into several simpler sub-tasks by LLMs, as the complex claim itself has intricate multi-hop relationships among inside entities, which is not conductive to evidence retrieval and veracity reasoning. In order to conform to practical application scenarios, we assume that the evidence can only be retrieved from some open domain sources, rather than annotated manually. Some common information retrieval method is first adopted to get the coarse-grained candidate evidence set from the open domain sources for each sub-task. Then LLMs are adopted to analyze complex relationships between candidate evidence and sub-task, and select the most useful evidence for each sub-task from candidate evidence. The refined evidence is further used to assist LLMs in executing each sub-task. The final prediction and explanation of the original complex claim are obtained after executing all sub-tasks. Experiments on the real-world datasets HoVER, demonstrate the superiority of ERECT over several baselines on complex claims verification.