Recent Advances in Applying Artificial Intelligence to Metadata Creation in the Social Science Japan Data Archive: Extracting Metadata from Social Surveys Using Large Language Models

To improve the effectiveness of metadata production in Japanese for social surveys, the Social Science Japan Data Archive (SSJDA) at the University of Tokyo's Institute of Social Science has built a metadata extraction method using OpenAI's API. Since 1998, SSJDA has manually created metadata, and since 2021 has followed the DDI-Codebook standard, but the increasing number of data deposits now requires more efficient methods.At last year’s EDDI poster session, we reported that Japanese metadata could be generated with a certain level of accuracy by designing OpenAI prompts using Python.We have made two key updates since then. First, we further developed the new GUI application by modifying the previous CUI version, improving accessibility for users. Second, we offer two options for users: cloud-based and local LLM. Although the local LLM does not yet match the quality of the cloud-based version, it has still produced highly consistent metadata with expert-created metadata, significantly improving processing time and cost efficiency compared to manual creation.SSJDA will continue to enhance development using more advanced LLMs and evaluate their integration into the metadata creation workflow. In the future, we plan to release the source code of the developed application on GitHub and consider distributing it to SSJDA depositors.