Quantitative Evaluation of Personality-Driven Short Dialogue Generation for Game NPCs Based on the Five-Factor Model

Personality-driven dialogue generation is essential for creating believable non-player characters (NPCs) in games. This study aims to (1) generate short NPC-like dialogue conditioned on predefined personality traits and (2) quantitatively evaluate whether the generated dialogue accurately reflects those traits. To achieve this, we propose a framework based on the OCEAN personality model for both controlled dialogue generation and systematic evaluation of personality consistency. We construct 32 personality configurations and generate responses to five scenario-based prompts using three models: Zephyr-7b, OpenChat-3.5-0106, and an Ollama-based OpenLLaMA-3B model. Personality consistency is evaluated using two complementary approaches: classification-based metrics (precision, recall, and F1-score) and score-based aggregation that measures alignment with intended personality traits. In addition, stability is introduced to quantify variability across multiple generated responses. The results suggest that the proposed framework supports a more structured comparison between high- and low-trait configurations within this controlled automated evaluation setting. OpenChat showed the highest performance in the automated evaluation, with F1-scores of 0.893 (high-trait) and 0.900 (low-trait), and the highest aggregated score of 340.94. Zephyr demonstrated strong stability (8.21) and consistent controllability, while the Ollama-based model showed lower consistency (F1: 0.715/0.743, score: 286.99) but substantially faster generation (0.57 s per response). Human validation on a representative subset supported the broad model-level tendency that OpenChat and Zephyr conveyed personality cues more clearly than Ollama, while the difference between OpenChat and Zephyr was less clear in human judgments.