Test Performance of Artificial Intelligence in the Chinese Social Work Certification Examination

Purpose: The study checked the validity of the entry-level Chinese Social Work Certification Examination in 2024 and tested the social work knowledge of artificial intelligence (AI) by exploring the performance of two systems—ChatGPT 4.0 and ERNIE Bot 4.0. Method: We applied zero-shot learning to the testing models, determined the correctness of their answers based on the reference books, and analyzed their reasons behind. Result: The results show that both AIs passed the exam with scores around 80/100. The AIs answered most questions correctly with adequate reasoning. The reasons for incorrect answers included wrong knowledge, logical fallacies, misunderstanding of question contents, and neglecting the test instruction, which reflected the hallucinations in large-language models. ChatGPT had better reasoning abilities, while ERNIE Bot was more familiar with local contexts and policies. Discussion: The findings support the validity of the exam and also confirm the ability of AI in using social work knowledge.