Can Large Language Models Be a Good Evaluator for Review-based Product Question Answering?

Large language models (LLMs) have demonstrated impressive performance on evaluating natural language generation tasks. Review-based Product Question Answering (RPQA) evaluation, which is a domain knowledge-intensive Question Answering (QA) evaluation, still largely relies on lexicon-based metrics and frozen embedding-based metrics. Those metrics fail if reference answers are absent. Despite some model-based metrics being learned from in-domain data for the RPQA task evaluation, little research has been done on using LLMs. Chain-of-Thought (CoT) is a state-of-the-art prompting method that has been proposed to induce LLMs to solve complex problems efficiently. The CoT reasoning steps can be resolved using the internal knowledge of LLMs, but this internal knowledge may be insufficient in domain knowledge-intensive RPQA evaluation settings. In this work, we explore the feasibility of leveraging LLMs as evaluators for RPQA evaluation. Specifically, we design a structured prompt template with one-shot or few-shot in-context learning, incorporating the CoT mechanism and the System 2 thinking process for RPQA evaluation. We call it as ExampLe-Enhanced CoT prompting (ELECT). We conduct a comprehensive study on various LLMs, including evaluation-oriented and general-purpose LLMs. Experimental results show that LLM-based evaluators can effectively evaluate RPQA. Upon integrating ELECT prompting into the demonstration examples in the prompt, we observe an improvement in the performance of the LLM-based evaluator. Code is publicly available at https://github.com/tonydhuang/elect-prompting.