What Does ChatGPT Know About Information Systems?

Large language models such as ChatGPT provide efficient access to a wealth of information. However, there are significant questions regarding the depth and quality of knowledge in any one domain. This paper focuses specifically on the information systems (IS) field and assesses ChatGPT’s knowledge. To analyze the extent and quality of information systems knowledge derived from queries to ChatGPT, we used over 3,000 queries from a broad range of exam and quiz questions. These queries were obtained from university courses and professional information system certification exams. The query topics are based on a framework for information systems education, with queries gathered from multiple faculty members at different universities. Our queries of ChatGPT with these questions finds that ChatGPT can answer 65% to 85% of information systems queries successfully, across each of the areas of the framework. ChatGPT performed well on essay, true/false and multiple-choice questions, with no statistical difference between its success rate on multiple-choice and true/false questions. We conclude that ChatGPT tends to perform approximately at the level of an average student but may not perform at a level sufficient to pass certain professional exams. ChatGPT’s knowledge is very broad, covering virtually all areas of the information system’s knowledge we identified, but not at an expert level.