METACURATE-ML: Generalising the extraction of questionnaires to DDI-Lifecycle

The canonical source for most large scale surveys remains for the past and the foreseeable future the ‘paper’ version of the questionnaire as a PDF. These contain the essential information on the fielding of the survey, the questions asked including response options, ordering and the logic associated with filtering to create DDI-Lifecycle metadata. However, there is no standard layout and formatting in these source documents which limits the scalability of heuristic programming approaches on OCR’ed text.The presentation will outline an approach to creating DDI-Lifecycle from semi-structured text in social science questionnaires using a combination of text-layout large language models (LLMs) and knowledge graph construction approaches, including methods from digital signal processing.