Quality Assessment and Enhancement of Generative AI Dialogue Summaries: ChatGPT vs. Human

This study compares Korean conversation summaries generated by ChatGPT-4o with human-written summaries to identify the strengths and limitations of AI summarization. Using a corpus from the National Institute of the Korean Language, 110 summary pairs were evaluated through BERTScore and expert review. Results show that AI summaries preserve core meanings and exhibit high grammatical accuracy but fall short in discourse structuring, paraphrasing, and reflecting speaker relationships. Based on these findings, five improvement strategies and a few-shot prompting method were proposed, leading to enhanced summary quality. This research offers empirical insights into the performance and refinement of generative AI in Korean summarization.