Development of an OpenAI GPT Response Evaluation Model Using Information Retrieval, BERTScore, and ROUGE Score (Case Study: Corporate Knowledge Base of PT Jinom Network Indonesia)

High operational loads on customer service and the risk of AI information hallucination pose major challenges to service automation at PT Jinom Network Indonesia. This research aims to develop and evaluate an OpenAI GPT response model integrated with an Information Retrieval (IR) system to manage a specific corporate knowledge base. The methodology employs the TF-IDF algorithm and Cosine Similarity to filter relevant documents, which are then used as the instructional basis for the GPT-3.5 model. A comprehensive system evaluation was conducted on 99 test data points (49 in-domain and 50 out-of-domain) using BERTScore and ROUGE-L. Rather than using a separate baseline model, the evaluation focuses on comparing the performance and discriminative ability of BERTScore and ROUGE-L in distinguishing valid (in-domain) from invalid (out-of-domain) questions. The results indicate that for in-domain questions, the system achieved an average BERTScore F1 of 0.8357 and an average ROUGE-L F1 of 0.5049. For out-of-domain questions, BERTScore decreased to 0.6752 (19.21%), while ROUGE-L decreased substantially to 0.1117 (77.88%). These results demonstrate that ROUGE-L has a stronger discriminative ability than BERTScore for distinguishing between in-domain and out-of-domain responses, based on the observed relative decrease in scores. The IR integration proved effective in restricting GPT responses to internal company data; however, the IR selection accuracy only reached 49.49%, indicating limitations of keyword-based retrieval in handling synonyms and semantic variations. In conclusion, the combination of BERTScore and ROUGE-L provides an objective evaluation framework for assessing chatbot response accuracy and determining its readiness for broader customer deployment.