LLM-assisted record linkage: A framework for official statistics

National statistical offices (NSOs) increasingly rely on record linkage to link census data, administrative sources, and survey responses. However, conventional string-similarity methods often struggle with free-text fields. To address these challenges, this paper systematically benchmarks modern open-source large language models (LLMs) against classic string-based comparators for record linkage. Building on these findings, this paper introduces a hybrid approach that retains well-established probabilistic frameworks yet integrates an LLM-based classifier for ambiguous record pairs. A Bayesian update is applied to combine the LLM's output with the prior probability, with the aim of reducing the burden on manual clerical review. The experiments show that selectively deploying open-source LLMs for the most uncertain pairs can significantly reduce manual effort by refining decisions through Bayesian updating. As NSOs must ensure transparency, explainability, and adherence to official statistical standards, this paper systematically addresses these concerns while evaluating the potential of LLMs for record linkage. Practical considerations including secure on-premises deployment, computational cost, human-in-the-loop review, and calibration are discussed to support responsible adoption in official statistics.

Authors

Institutions

Publication Details

Journal
Statistical Journal of the IAOS
Published
2026-02-26
DOI
https://doi.org/10.1177/18747655261422068
Primary Topic
Data Quality and Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

LLM-assisted record linkage: A framework for official statistics

Hanan Ather
Statistical Journal of the IAOS
Data Quality and Management
article

LLM-assisted record linkage: A framework for official statistics

Hanan Ather
article en

Abstract

National statistical offices (NSOs) increasingly rely on record linkage to link census data, administrative sources, and survey responses. However, conventional string-similarity methods often struggle with free-text fields. To address these challenges, this paper systematically benchmarks modern open-source large language models (LLMs) against classic string-based comparators for record linkage. Building on these findings, this paper introduces a hybrid approach that retains well-established probabilistic frameworks yet integrates an LLM-based classifier for ambiguous record pairs. A Bayesian update is applied to combine the LLM's output with the prior probability, with the aim of reducing the burden on manual clerical review. The experiments show that selectively deploying open-source LLMs for the most uncertain pairs can significantly reduce manual effort by refining decisions through Bayesian updating. As NSOs must ensure transparency, explainability, and adherence to official statistical standards, this paper systematically addresses these concerns while evaluating the potential of LLMs for record linkage. Practical considerations including secure on-premises deployment, computational cost, human-in-the-loop review, and calibration are discussed to support responsible adoption in official statistics.

Statistical Journal of the IAOSVol. 42(1)
Statistics Canada (CA)
Peace, Justice and strong institutions
Openalex Percentile: Top 48%
Data Quality and Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.