Identity Resolution

A birth year can be corrected. A merge cannot, in any practical sense: it repoints a dozen foreign keys, retires an identifier that may already be in print, and destroys the distinction between two records that might have been genuinely different people.

Section 1One person, many arrivals

A political career generates records in many places. The same person may appear as a candidate in six elections, as a party branch chair, and as a legislative committee member — and in the Taiwan Politics database each of those is a separate row in a separate table, arriving from a separate source, often loaded years apart.

Each arrival needs to point at a person. The database's person table is officials, and everything else references it.

So the question keeps recurring: when a new party-leadership record arrives naming 吳美玲, is that the 吳美玲 already in officials, or a different one?

Three ways to answer it, and only one is honest.

The third is right. It asserts exactly what the source supports, it produces something referenceable and analysable, and its failure mode is a duplicate that the audit layer can detect and route to a person.

This is the standard record-linkage pattern: mint, then adjudicate, then consolidate. Its cost is that you deliberately create duplicates. Its benefit is that you never silently assert an identity you have not established.

Section 2Hub and spoke

officials is the person hub. Seven relations reference it.

Two asymmetries on that diagram matter.

Candidacies are already resolved. All 94,217 of them point at an official, with zero unresolved cases. Candidacy data arrived with identifiers; the leadership and family data did not. So minting applies to the leadership and family populations, not to candidates.

family_records references the hub twice. The ego is the person the record is about; the alter is their relative. This is not symmetric, and it drives a specific exclusion: do not mint provisional officials for alters. A person named as someone's relative is not thereby an official. Minting for alters would flood the hub with people who never held office and would defeat the referential rule whose entire job is to match family mentions to existing officials without creating new ones.

Section 3Three operations

Identity resolution is three distinct operations. Conflating them is the commonest way this goes wrong.

The architecture contract keeps minting out of the audit layer — intake is a separate procedural-control layer — and keeps consolidation out of the evaluator: a same_entity decision remains pending until the approved consolidation completes and revalidation passes.

Why insist on the separation? Because each has a different failure mode and a different reversibility. Minting too eagerly creates work. Adjudicating wrongly creates a false decision that can be reopened. Consolidating wrongly destroys a distinction, and there is no reopening that.

Section 4Candidate generation

With 50,902 officials, comparing every pair means about 1.3 billion comparisons. Nobody does that. Instead you use a candidate-generation key — also called a blocking key — to propose a small set of plausible matches.

For Taiwanese political figures the obvious ingredients are name, birth year, and sex. But how they are combined matters enormously, and the architecture contract issues a warning that is easy to skim past:

“Name/birthdate or another comparison key is a candidate-generation key, not proof that two people are identical and not a formal uniqueness constraint.”

The key proposes. A researcher decides. Conflating those turns a heuristic into an assertion.

The outcomes available to that researcher are four, not two:

different_entities is the outcome people forget to build, and it is the most common one in practice. Most same-name pairs are different people. Recording that explicitly — rather than leaving the pair in a queue forever — is what stops the queue from filling with cases that have already been settled.

Section 5The transitivity trap

Now a subtlety with real consequences.

It is tempting to write the key as: same name, same sex, birth years within ±2. The tolerance handles sources that disagree slightly, which they frequently do.

But ±2 is not transitive, and that breaks things in two different ways depending on how you group the results.

Neither option is acceptable as stated. Pairwise adjudication lets a researcher record A=B, B=C, and A≠C — a set of decisions that cannot all be true. Grouping by transitive closure fixes the coherence problem and replaces it with an unbounded one: with a common name, the chain can absorb an entire block of unrelated people into a single unreviewable group.

And these are not hypothetical sizes. An earlier scan of this database found 5,073 same-name pairs with conflicting birth years and hit its 2,000-record cap before finishing.

Section 6Blocks or components

Two ways to form the group a researcher adjudicates. This project has both available, which turns out to matter.

Connected components

Treat records as nodes and pairwise matches as edges; compute connected components. This is what the lab's existing batching service does — its grouping method is literally named connected_components_by_shared_record.

It solves non-transitivity, because the researcher sees the whole component and partitions it in one pass. Its weakness is exactly the chain above: group size is unbounded.

Blocking key

Group by an exact key — say (normalized_name, sex_code) — and display birth year as context rather than encoding a tolerance in the predicate.

Group size is now bounded by the key. Chains cannot form. The ±2 tolerance moves out of the predicate and into the researcher's judgement, where it belongs, because a human looking at a block of five 黃建宏 records with birth years 1961, 1963, 1971, 1972, and 1988 can partition them far better than any tolerance parameter.

For this rule, blocking is the better choice — not because components are wrong in general, but because a loose tolerance over a small name space makes chaining a certainty rather than a risk.

Whichever method is used, the grouping_version must be pinned. Changing the name normalisation changes the groups — and a decision recorded against one grouping is not automatically valid under another.

Section 7What a merge must do

A researcher decides that officials 3417 and 3418 are one person. Now the expensive part.

The redirect

Step 4 is the one that distinguishes a research database from an application database, and it is worth dwelling on.

Official 3418 does not simply cease to exist. Its identifier may already have been:

• cited in a published dataset release;
• referenced in a paper's replication files;
• recorded in this system's own evidence rows, attached to earlier decisions;
• bookmarked by a collaborator.

If 3418 returns nothing after the merge, every one of those breaks — and the system's central promise, that a value can be traced and checked, breaks with it.

So the retired identifier is retained and resolves to its survivor. The record is gone as an entity; the identifier remains as a pointer. This is the same discipline that makes a DOI durable, applied at row granularity.

Section 8Deciding before merging

If consolidation is that expensive, does adjudication have to wait for it?

No — and the architecture already permits the alternative. Recall that same_entity “records the decision but remains pending until the approved consolidation completes.” Decision and resolution are separate dimensions, exactly as Chapter 6 set out.

So a rule can ship in a staged form:

Stage 1 is genuinely useful on its own. The queue gets worked, the adjudications are real and recorded, and the merge service is built against a known backlog rather than a hypothesis about one. The three-dimension state model is what makes this coherent rather than a fudge: a pending same_entity is not a half-finished task, it is an accurately described one.

Section 9Decisions go stale

A final complication, specific to identity and easy to miss.

This project sequences its rules deliberately: birth completeness first, then sex completeness, then duplicate officials. The reason is that the first two rules improve exactly the fields the third one blocks on. Researchers spend weeks raising the quality of names, birth years, and sex values, and identity matching then generates better candidates with fewer false pairs.

But it follows that the identity rule's population keeps moving while it is being worked. Consider:

1. Two officials share a name. Both have unknown birth years. A researcher examines them, finds no distinguishing evidence, and records different_entities.
2. Weeks later, a different researcher working the birth-completeness queue establishes a birth year for one of them.
3. It matches the other's newly established birth year exactly.

The earlier decision was reasonable when it was made and is now probably wrong. Nothing about it looks wrong. It will sit in the database as a confident, obsolete answer unless something notices.

That something is the revalidation_trigger_set. For an identity rule it must include official_name, birth_year, birth_status, sex_code, and sex_status — every field the candidate-generation key touches. When one changes, affected decisions become stale and the collision reopens.

The contract provides for this: a material subject revision may make any earlier decision stale and reopen the collision. But it has to be wired, and it has to be tested — with a fixture that resolves a collision, mutates a birth year, and asserts the reopen actually happens. That test is the difference between a system that stays correct and one that merely started correct.

With identity settled, the canonical data is as good as the lab can currently make it. The last chapter turns the result outward: derived views, versioned datasets, codebooks, and what it takes to make a download citable.