Derivation, Release, and Download

Everything up to this point has been about making the lab's own data trustworthy. Release is about making that trustworthiness legible to someone who was not there — which turns out to require almost as much discipline as producing it did.

Section 1Computed, not collected

Ask the Taiwan Politics database a research question: at what age did party leaders typically first hold elected office?

Nothing in the canonical tables answers that. officials holds birth information. candidates holds candidacies. party_leaders holds appointments. The answer requires assembling one person's entire career from three tables, ordering it, finding the first elected position, and differencing against a birth date.

That assembly is the derivation layer: tables and views whose contents are computed from canonical rows rather than collected from sources.

Taiwan already has a substantial one. A resume-event view produces 95,055 rows — every career event for every official, normalised into a common shape. A career-sequence view produces 50,902 — one ordered career per official.

Notice what derivation depends on. A career sequence is only meaningful if each candidacy and appointment points at the right official — so the derivation layer inherits every identity decision from Chapter 8. And age at first office is only computable because Chapter 5 produced a typed birth_date rather than leaving 民國45年 in a text column.

The derived layer is where the earlier work pays off, and also where it is audited in practice: an implausible career sequence is usually a symptom of an unresolved identity or a mis-parsed date upstream.

Section 2Never edited, only recomputed

One rule governs this entire layer:

Derived values are never edited. They are recomputed.

If a career sequence is wrong, the fix is upstream — in the canonical rows — followed by recomputation. There is no resolution form for a derived value and there must never be one.

The temptation is real and specific. A researcher notices that an official's career shows a legislative appointment beginning before their first candidacy, which is implausible. The fastest fix is to edit the sequence. The correct fix is to work out why: a mis-parsed date, a mis-assigned identity, a genuinely unusual career. Each of those has a different upstream repair, and only one of them is a data error at all.

Editing the derived value hides which. It also guarantees the edit is destroyed the next time the view is recomputed — or worse, is not destroyed, because someone materialised the view, and now the derived layer permanently disagrees with the canonical tables it claims to summarise.

Section 3View, materialized, or table

Three forms, with different costs.

Prefer a plain view. Reach for materialisation only when a measured query is too slow — not in anticipation, because materialisation buys speed with a new class of bug: content that is silently out of date.

If you do materialise, two obligations follow. The contract requires an explicit refresh and staleness contract, and it is emphatic that materialized views are never the authoritative decision history. Practically: any screen where a researcher can act on a materialized value must display when it was last refreshed. A stale number a person can act on is worse than a slow query.

Section 4Five commitments

Now the release itself. A CSV export is not a research output. A curated dataset is a CSV plus five commitments, and the difference is whether anyone can cite it.

The Taiwan Politics About page currently promises all five and delivers none. Its text reads, in part: “Current metadata status: Development... The first release notes will add a version identifier, substantive update date, change summary, expected update cadence, and archive policy” — and, against the codebook, simply “Not available yet.”

That page is being honest, which is the right behaviour for an unreleased database and is worth more than a confident page that overstates. But it is a gap, and closing it is what this section is about.

Section 5Versioning

Four rules.

A released version is immutable. Republishing different bytes under the same version breaks every citation of it. If someone's regression table was run against version 1.0, version 1.0 must still be the thing they ran it against.

Corrections increment. 1.0 → 1.1 for a data correction; → 2.0 when the field set or unit of analysis changes. A reader can then tell at a glance whether their analysis code will still run.

Archive superseded versions; never delete them. A paper published against 1.0 must remain reproducible after 1.1 ships. Deleting an old release does not remove the papers that cite it; it only makes them uncheckable.

Record which rule versions governed the release. This is the one that this architecture makes possible and that most pipelines cannot offer.

That fourth point is the payoff for all the machinery in Chapters 6 and 7. Because rule versions are immutable and checksummed, a dataset can state:

Compare that with the usual claim: “the data has been cleaned.” Unfalsifiable, and quietly implies uniform quality across every column. The table above tells a reader exactly which fields were examined, under what standard, and — in the last row — which were not examined at all.

Admitting the last row is what makes the first three believable.

Section 6The codebook

A codebook defines, for every published field: its name, type, and unit of analysis; the complete value vocabulary for coded fields; the missing-value convention; the audit rules that governed it, by code and version; and known gaps.

Missing values are the hard part

This is where Chapter 5's status vocabulary earns everything it cost.

The difference is not documentation quality. It is whether the reader can model the missingness at all.

Under the flattened version, an analyst has one undifferentiated pool of nulls. Under the preserved version they can see that missingness in unknown rows is a property of the historical record — likely correlated with era, region, and prominence, and therefore itself a finding — while missingness in not_collected rows is an artefact of scope with no substantive interpretation. Those require completely different handling, and a codebook that cannot distinguish them invites a mistake.

Generate it; do not write it

A hand-written codebook drifts from the database within one release cycle. Someone adds a status value, or a rule version changes, and the prose does not follow.

Every piece of a codebook already exists as machine-readable data:

So generate the codebook from those sources. The contract already establishes the principle for rule documentation — do not independently maintain conflicting SQL, Python, Markdown, and CSV rule definitions — and it applies identically here.

The only genuinely hand-written parts are the ones that require judgement: the unit of analysis, the known gaps, and the collection scope. Those are short, and they are the PI's to write.

Section 7Disclosure

Not everything the lab holds should be published. Four access levels, applied per field rather than per table:

Two rules follow, and the second is a genuine trap.

Enforce at the queryset or serializer boundary, never in the template. The database layer's own baseline requires it: public pages suppress restricted fields and unpublished records at the queryset/API boundary. A field hidden only in a template is still in the response, still in the JSON export, and still in anything that reuses the queryset.

withheld as a data status is not the same as researcher-level access. birth_status = 'withheld' means the lab knows the value and has decided not to publish it. That is a publication decision about a specific value. Researcher-level access is an access level for a whole field. They are orthogonal, and a withheld value must not leak through an export authorised for a different reason.

The disclosure matrix is one row per published field, recording its level, the rationale in a line, and what happens when the field's status is withheld. It does not yet exist for any schema in this project and is a prerequisite for the first public download.

Sex and birth data on named political figures are the cases to think hardest about. Much of it is a matter of public record for elected officials; some is not. The matrix is where that judgement is recorded once, rather than remade inconsistently every time someone exports something.

Section 8The download layer

The mechanics already exist, and their design is worth studying.

The key design choice is that this is registry-driven. A downloadable resource is a metadata registration, not a new view. Adding a dataset means registering it; it does not mean writing an exporter.

That is why the Taiwan About page can report, accurately and without anyone hand-editing it, that “Configured Explore tools currently include Positions Roster” — the page is reading the registry.

Section 9The release checklist

Where this series ends

Nine chapters ago we had four rows in a spreadsheet: an era-year birth date, an unexplained null, and two men who might be one man.

None of those problems has been solved by software. 民國45年 was converted by a person who read a bulletin. The null was classified by a person who searched and did not find. The two 黃建宏 records were adjudicated by a person who weighed the evidence.

What the system did was narrower, and it is the honest summary of what a data management system is for:

• It preserved the original, so that no decision was foreclosed by an earlier convenience.
• It detected which records needed attention, under a standard that is written down and versioned.
• It routed them to people, in a tractable order, with the context needed to decide.
• It recorded what they decided, on what basis, under which version of which rule.
• It noticed when a decision went stale.
• It published the result in a form that says what was checked and what was not.

That is the whole job. The database does not know anything about Taiwanese politics. It knows which questions have been asked, of which records, by whom, and what they answered — and it makes that legible to someone who was not in the room.

None of this is ever finished. New rules are added a few at a time, as researchers become ready to work them. New data arrives and needs intake. New databases arrive and need the whole sequence again. The system is not a project with a completion date; it is a practice, and its measure is not how clean the data is but how well a stranger can tell how much to trust it.