Assessing the Reliability of Large Language Models in Generating Synthetic and Representative Datasets for International Study Program Recommendations

The rise of generative AI, particularly Large Language Models (LLMs), has revolutionized the tech industry, research, and society, sparking excitement about their potential to automate tedious tasks while also raising concerns over their reliability, ethical implications, and misuse. This study investigates the reliability of popular open and free LLMs, such as ChatGPT, Gemini, and DeepSeek, in generating accurate, well-supported datasets in the field of education. Our research is rooted in a project where a web application was developed to help students find suitable international study programs. The project required extensive data collection, including tuition fees, university rankings, healthcare costs, general cost of living, etc. The sheer difficulty of finding and accumulating that data, which was oftentimes poorly documented, prompted us to examine LLMs' performance in replicating these datasets. We evaluated their outputs by calculating the coefficient of determination between synthetic and fact-checked real data. Results demonstrate that LLMs can generate representative data, though occasional deviations raise questions about their limitations and the complexity of prompt engineering. This research sheds light on the practicality of LLMs in automating data-intensive tasks while highlighting their potential and limitations, offering valuable insights into the broader capabilities of generative AI.