Probabilistic Model Streamlines Large-Scale Data Merging

Merging large administrative data sets is challenging due to missing identifiers and inaccurate records. This paper introduces a new algorithm for probabilistic record linkage that handles these issues efficiently at scale.

Data & Methods: We developed a fast, scalable algorithm using a canonical probabilistic model approach. The method accommodates millions of observations while accounting for:

• Missing data
• Measurement error
• Auxiliary information integration
• Uncertainty adjustment in post-analysis

Simulation Studies: Our algorithm was tested extensively through realistic scenarios to ensure reliability.

Real Applications: Case studies demonstrate its use in merging campaign contribution records, survey datasets, and voter files. An open-source implementation is available for researchers.