Fellegi & Sunter (1969) - A Theory for Record Linkage
Key Insights
- Fellegi and Sunter provide the mathematical foundation for record linkage, establishing a probabilistic framework for determining whether two records refer to the same entity based on agreement patterns of common identifying fields.
Extracted Variables
No quantitative variables detected.
Study flow (PRISMA-style)
Machine-extracted from the synthesis; review before citing.
Edit on GitHub — registry.json
Background
Before Fellegi and Sunter, record linkage was practical folklore: clerks compared fields, applied rules of thumb, and disagreed. Their 1969 paper in the Journal of the American Statistical Association converted the problem into probability theory — and every entity-resolution system since, from census bureaus to deduplication engines, descends from it.
The Formulation
Two records either refer to the same entity (match) or different entities (non-match). The paper defines a comparison vector: the pattern of agreement and disagreement across common identifying fields. The question becomes statistical: given a comparison vector, which decision — match, non-match, or possible match (requiring review) — minimizes expected error at controlled levels?
Deep Dive
Under conditional independence of fields given true status, the optimal decision rule uses weights that are log-likelihood ratios: for each field, the m-probability (agreement among true matches) over the u-probability (agreement among true non-matches). Summing field weights yields a total score, and the paper's theorem gives a pair of thresholds (lower for non-match, upper for match) that respect user-specified error bounds, leaving a middle band for clerical review. The practical art is estimating m and u without labeled data — later work applies the EM algorithm — and managing blocking to avoid comparing every record pair. Modern implementations, including the dedupe library, directly implement this machinery.
Why It Matters
Probabilistic record linkage is the theoretical foundation of identity resolution, and identity resolution is the foundation of KYC, sanctions screening, and customer-360 programs. Understanding the weight model explains why fields are scored rather than binary-matched, and why threshold tuning is an accuracy-versus-recall decision.
Key Takeaways
- Weights are log-likelihood ratios — rare agreements signal more than common ones.
- Thresholds are error-budget controls: the review band is where precision and recall are negotiated.
- Blocking is mandatory at scale; choose keys (name, dob, jurisdiction) that preserve matches without exploding pairs.