A Novel Summarization Framework based on Reference-Free Evaluation of Multiple Large Language Models

Recently many researchers study the problem of summarizing the abstraction of medium news utilizing Large Language Models (LLMs). However, the single model usually produces results with some flaws. This paper studies how to make fusion of results from multiple LLMs. Our main contribution is summarized as follows. First, we trained multiple summarization models from multiple LLMs, including Meta-Llama-3-8B-Instruct, Qwen2-7B-Instruct, and GLM-4-9b-chat. Second, we design a new reference-free evaluation metric, which could make fusion of results from multiple big models. Third, we make experiments to evaluate the performance of our proposed framework. The experiments show that the fusion solution multiple models could produce better results than single model. The summarization result produced by this fusion solution is more consistent with human evaluation terms of coherence, consistency, relevance, fluency.