Automated Question Answering for Thai Visa Services using GPT-4: System Architecture, Custom Dataset, and Performance Evaluation

Thailand’s immigration system includes various visa types, such as Tourist (TR), Business (Non-Immigrant B), Education (Non-Immigrant ED), ED Plus, Media (Non-Immigrant M), Official (Non-Immigrant F), Dependent/Spouse (Non-Immigrant O), SMART, Transit, Long-Term Resident (LTR), Retirement (O-A), and Long-Stay Retirement (O-X), each governed by distinct and frequently updated policies. This complexity causes confusion and difficulties for foreign nationals in understanding required documents, eligibility criteria, and policy changes, as shown by a survey involving 75 respondents in Bangkok. Without scalable and accurate solutions, these challenges can reduce service efficiency and public trust. To address this, the study employs GPT-4, a Large Language Model (LLM), optimized with prompt engineering and few-shot learning to automatically answer visa-related questions with accurate, clear, and policy-compliant responses. A simulated dataset of 1,600 question-answer pairs was created from official Thai immigration policies. GPT-4’s performance was systematically evaluated using automatic metrics (BLEU, ROUGE-L, METEOR) and human assessments of accuracy, clarity, and user satisfaction, providing a robust methodological basis for evaluating its effectiveness in public service. Results show strong performance across most categories, with the highest BLEU scores observed in Tourist (0.82), Business (0.81), and Long-Term Resident (LTR) (0.78) visa-related queries. Corresponding peak ROUGE-L scores were 0.86 for Tourist, 0.84 for Business, and 0.81 for Long-Term Resident (LTR) visas, while METEOR scores peaked at 0.79 (Tourist), 0.78 (Business), and 0.76 (Long-Term Resident (LTR)). The system achieved an overall BLEU score of 0.75, ROUGE-L of 0.79, and METEOR of 0.73. Human evaluators rated GPT-4's responses with an average score of 4.5 for accuracy, 4.6 for language clarity, and 4.4 for user satisfaction (on a 5-point scale). A Pearson correlation of 0.78 between BLEU and human-rated accuracy indicates high alignment between automated and human evaluation. These results highlight GPT-4's potential in enhancing public-facing services such as immigration services by providing accurate, clear, and policy-aligned responses. Future work will focus on multilingual support, real-time policy updates, and deployment in live service environments.