An End-to-End Workflow for Building Longitudinal Policy Measures From Government Web Sources Using Historical Reconstruction and LLM-Based Extraction

Background: Longitudinal policy research requires measures of what policy applied, when, and with what provisions, yet government sources are organized and preserved in ways that can make historical policy difficult to reconstruct. This study developed and evaluated an end-to-end workflow for constructing longitudinal policy measures from government web sources.Methods: The workflow was applied to NHS England’s Additional Roles Reimbursement Scheme (ARRS), a national policy specifying which primary-care workforce categories are eligible for reimbursement, from 2019/20 to 2026/27. It combined historical document recovery, variable specification, automated extraction against a human-coded reference standard, and longitudinal measure construction. Keyword, deterministic, and large language model (LLM) methods were compared across eight development and six later evaluation versions. Three simplified measurement strategies were compared with the author-verified longitudinal reference. Exploratory source recovery assessments were also conducted in Spain, France, and Australia.Results: Historical reconstruction identified 14 substantive document versions across eight policy years, including within-year changes in reimbursement eligibility, amounts, and conditions. Reimbursable primary-care workforce categories increased from 5 to 27, while the current NHS England landing page provided access to documents from only one policy year. On the six later evaluation versions, the LLM matched all scored reference-standard targets for eligibility status, effective dates, and reimbursement amounts, although eligibility conditions remained more difficult to extract. Simplified measurement strategies changed the reconstructed policy history. Using only the current specification could not recover historical eligibility, one specification per policy year assigned 96 eligible category-months (7.3%) too early, and keyword coding produced 138 category-month errors (10.5%).Conclusion: Valid longitudinal policy measurement requires historical reconstruction, explicit variable and dating rules, evaluation on later documents, and human verification. For health-politics research, this approach can support measures of changing governmental choices about eligibility, financing, conditions, and timing while distinguishing formal policy from implementation and outcomes.