Prompt Scoring System for Dialog Summarization

Recent advancements in language processing have demonstrated the advanced capabilities of language models. Particularly noteworthy is the heightened prowess of pre-trained large language models in tackling tasks that were a real challenge a few years ago, such as the abstractive summarization of dialogs. An approach to generating summaries involves engineering prompt templates. The easiest way would be by using a static prompt, but it can lead to unreliable outcomes for different classes of dialogs. We implemented a scoring system to enhance the performance of a few-shot training. This involves constructing finely tuned prompts composed of dialog samples with the highest scores. The scoring process is grounded in a set of heuristics that specifically assess the structure and content of the dialogs. The use of the scoring system resulted in enhanced ROUGE scores and positive evaluations from human assessors. These promising results were consistently validated across all three large-scale datasets used in the testing phase.