Can ChatGPT Effectively Handle Economics Questions? Evidence from Hong Kong

Economics subjects extensively adopt problem sets as a key assessment component. As universities open door to Generative Artificial Intelligence tools like ChatGPT, students' misuse may prompt academic integrity issues, thus impairing the fairness of assessment results in teaching. Using a rich problem set, we find that ChatGPT has substantial limitations in handling graphical questions. Its effectiveness is significantly compromised when attempting complex questions involving both graphical (or contextual) and numerical analysis. Our findings offer support to more extensive use of a combination of numerical and graphics questions in assessment tasks. The findings also prompt an economics instructor to adopt a balanced distribution of questions when developing the problem set.