Provenance

A value in a research database is worth what its provenance is worth. A birth year with a citable source, a retrieval date, and a content digest is a finding. The same birth year with nothing behind it is a rumour with a foreign key.

Section 1How do you know?

Suppose a paper reports that Taiwanese party leaders in the 1990s were on average four years older than legislative committee chairs. A reviewer asks where the birth years came from.

There are several possible answers, and they are not equally good.

• “From our database.” This is not an answer. It relocates the question.
• “From Central Election Commission bulletins.” Better, but unfalsifiable at the level of any particular value. Which bulletin? Was this leader's birth year in a bulletin, or filled in from elsewhere?
• “Of the 412 leaders in the sample, 380 birth years are attributed to specific CEC bulletins with retrieval dates and content digests; 22 come from legislative gazettes; 10 were resolved by a researcher from a named biographical reference and carry a recorded decision.” That is an answer. It is also checkable, which is the point.

Producing the third answer is not a matter of writing it down more carefully. It requires the database to have been built so that the information exists at all. That is what this chapter is about.

Section 2Four concepts

Four things are involved, and they are routinely conflated into one vague notion of “provenance.” Keeping them apart is the whole discipline.

The first three describe where data came from. The fourth describes why a person made a decision. They can cite the same document. They are not the same thing, and §6 explains what breaks if you merge them.

Section 3The source

A source is a persistent, normalised record of a document or dataset. Not a URL in a text column — a first-class entity, created once and cited by many rows.

class Source(models.Model):
    """One normalized external or researcher-created source artifact."""

    class SourceType(models.TextChoices):
        GOVERNMENT_DATASET = "government_dataset", "Government dataset"
        WEBPAGE           = "webpage",            "Webpage"
        ARCHIVE_FILE      = "archive_file",       "Archive file"
        PUBLICATION       = "publication",        "Publication"
        INTERVIEW         = "interview",          "Interview"
        RESEARCHER_NOTE   = "researcher_note",    "Researcher note"
        OTHER             = "other",              "Other"

    source_id       = models.UUIDField(primary_key=True, default=uuid.uuid4)
    source_type     = models.CharField(max_length=24, choices=SourceType)
    authority       = models.TextField()
    title           = models.TextField()
    canonical_url   = models.URLField(max_length=1000, blank=True)
    archived_uri    = models.TextField(blank=True)
    published_at    = models.DateField(blank=True, null=True)
    accessed_at     = models.DateField(blank=True, null=True)
    version_label   = models.TextField(blank=True)
    content_digest  = models.CharField(max_length=128, blank=True)
    citation_text   = models.TextField(blank=True)
    language_code   = models.CharField(max_length=16, blank=True)
    jurisdiction_code = models.CharField(max_length=16, blank=True)

authority is who stands behind the document, distinct from where it happens to be hosted. The Central Election Commission is the authority for a CEC bulletin whether you read it on their site, in a university archive, or as a PDF a colleague emailed you. Treating the host as the authority is a common and consequential slip.

published_at and accessed_at are deliberately separate. A bulletin published in 1998 and retrieved in 2024 has a twenty-six-year gap in which its hosting, formatting, or availability may have changed. For web sources the gap matters even more: a page that said one thing in 2024 may say something else now, or may be gone.

content_digest is a hash of the retrieved bytes. It is what converts “we cite this page” into “we cite this exact content, and here is how to tell whether it has changed.” For a stable government PDF it is a formality. For a webpage it is the difference between a citation and a hope.

version_label handles the case where an authority reissues a corrected edition. Two sources, same authority, same title, different versions — and a canonical value attributed to one of them, not vaguely to both.

Section 4The locator

A source identifies a document. A locator identifies a position inside it.

The distinction earns its keep because sources are large. A CEC bulletin covering a municipal election may run to hundreds of pages listing thousands of candidates. “This name came from the 1998 bulletin” is nearly useless for checking; “page 147, table 3, row 22” can be verified in a minute.

Locators are stored as text on the evidence link rather than as a structured type, because their shape depends entirely on the source:

Free text is a compromise, and worth stating as one. A structured locator would be machine-checkable. But a schema general enough for both a PDF page and a CSV cell ends up as an untyped key-value bag anyway, with the added cost of pretending otherwise. Text plus a documented per-source-type convention is the honest version.

Section 5Record evidence

Record evidence is the link: this source, at this locator, supports this canonical record.

class RecordEvidence(models.Model):
    """One source attribution for one canonical Taiwan record revision."""

    class EvidenceRole(models.TextChoices):
        PRIMARY_SOURCE = "primary_source", "Primary source"
        CORROBORATING  = "corroborating",  "Corroborating"
        DERIVED_FROM   = "derived_from",   "Derived from"
        IMPORT_SOURCE  = "import_source",  "Import source"
        OTHER          = "other",           "Other"

    class AttributionMethod(models.TextChoices):
        INTAKE           = "intake",           "Intake"
        RETROACTIVE_BULK = "retroactive_bulk", "Retroactive bulk"
        RESEARCHER       = "researcher",       "Researcher"
        DERIVED          = "derived",          "Derived"

    record             = models.ForeignKey("audits.Record", ...)
    source             = models.ForeignKey("provenance.Source", ...)
    evidence_role      = models.CharField(choices=EvidenceRole)
    source_locator     = models.TextField()
    attribution_method = models.CharField(choices=AttributionMethod)
    transform_version  = models.CharField(max_length=100)
    record_revision    = models.CharField(max_length=128)
    is_current         = models.BooleanField(db_default=True)
    supersedes_record_evidence = models.ForeignKey("self", null=True)

Three fields here are doing something subtle.

evidence_role distinguishes a primary source from corroboration. Two sources agreeing on a birth year is a stronger claim than one source asserting it — and that strength is only visible if the database records which is which. import_source marks the weakest case honestly: this value arrived in a bulk load and its ultimate origin is the load, not a document anyone has cited.

attribution_method records how the link was made, which is a different question from what the source is. An attribution created at intake, one created by a retroactive bulk backfill, and one created by a researcher working a violation have very different reliability. A bulk backfill that matched 139,146 rows by a deterministic key is defensible, but it is not the same as a person who opened the document and looked.

transform_version records which transformation produced the canonical value from the source value. When you later discover that a parser mishandled a date format, this field is how you find every value that parser touched.

Note also the target: record, a foreign key to audits.Record — the stable identifier introduced in Chapter 2. Not the official's integer primary key. Evidence attaches to the durable identity of a record, so that it survives merges and corrections. And record_revision pins which version of that record the evidence was attached to.

Section 6The separation

Now the central point of the chapter.

Record evidence describes where data came from, and exists whether or not any rule is satisfied. Violation evidence describes why a researcher made a decision, and exists only for violations.

Both can cite the same document. That is not duplication — it is two genuinely different facts about the same bulletin.

What breaks if you merge them

Suppose you store one “evidence” table for both. Four questions become unanswerable.

• “Which values were imported and which were adjudicated?” Both now look identical: a row pointing at a source. The distinction between data the lab loaded and data a specialist established has been erased.
• “What is the provenance of this dataset?” The answer is now contaminated by audit history. A field checked five times by researchers appears better-sourced than an identical field checked once, even when both trace to the same bulletin.
• “What did this researcher actually decide?” Their citation is now indistinguishable from an import attribution made by a script.
• “Can we archive the audit history?” No — archiving it would strip the provenance from the canonical data, because they are the same rows.

That last one is the practical killer. The lab's archive cutover, which retires obsolete rules while preserving their history behind read-only routes, is only possible because retiring audit history does not touch the attribution of a single canonical value.

Section 7Supersession

Provenance changes. A better source turns up; an earlier attribution proves wrong; a bulk backfill is superseded by a researcher who actually opened the document.

Record evidence handles this with two fields visible in §5: is_current and supersedes_record_evidence, a self-referencing foreign key.

An attribution is never deleted or edited. A new one is created, pointing back at the one it replaces, and the old one's is_current becomes false. The result is a chain, which supports two different readings of the same data:

• Current view — filter on is_current. This is what a codebook and a dataset release use.
• Historical view — follow the chain. This is what you need when asking why a value changed, or auditing whether a bulk backfill was sound.

The same append-only pattern recurs throughout the system: audit decisions supersede rather than overwrite; actions are an immutable ledger; archived rules are deactivated rather than deleted. It is the natural shape for a system whose subject matter is what was believed, by whom, on what basis, and when.

Section 8The cost of doing it late

Provenance is the step most often deferred. It is invisible in the product, nothing breaks without it, and there is always something more urgent. This project deferred it, which makes it a useful case study in what deferral costs.

The Taiwan data was loaded with immutable raw capture — so nothing was lost — but without normalised sources and record-evidence links. Those are now being reconstructed retroactively, and the reconstruction involves:

Every one of those steps is work that writing an attribution row at load time would have avoided. Step 3 is the painful one: the canonical values have moved since the snapshot, so reconciliation is no longer a join but a reconstruction of history.

And note what step 5 concedes. Some attributions cannot be recovered — not because anyone was careless, but because the information about which of several plausible sources produced a given value existed only in a researcher's head in 2023.

The data is in, and we can say where it came from. The next chapter is about making it usable without destroying it — the legacy-field pattern, and the six-word vocabulary that lets a researcher say something true about every absent value.