Step 5 · Validation

Content analysis has a mature answer to the question "how do we know the coding is right?": have two or more coders code the same material independently and measure how often they agree beyond chance. The workflow uses that answer twice. Three AI coders are compared with one another, which tells you whether the codebook is applied consistently. Then humans are compared with the AI, which tells you whether it is applied correctly. The first is cheap and runs before anyone in the room opens a spreadsheet; the second is the one that counts.

Section 1Why validate, and how

Three statistics are used here. Cohen's κ measures chance-corrected agreement between two coders on one categorical variable; it is the familiar figure and is reported for every pair. Fleiss' κ extends it to several coders at once. Krippendorff's α does the same and copes with missing values and other levels of measurement; it is the statistic most content-analysis journals now expect. The bands conventionally applied to α are: at or above 0.80, reliable; between 0.67 and 0.80, tentative conclusions only; below 0.67, revise the instrument.

In the full-scale application reported in the presentation, human coders agreed with the AI at κ ≈ 0.80 on high-confidence rows, against κ = 0.81 between the human coders themselves, and 92 percent of AI-coded relevant rows were confirmed relevant by humans. Agreement fell to about 0.5 on medium-confidence rows. That pattern, high where the coder said it was sure and lower where it did not, is what you want to see: it means the confidence score carries information and can be used to decide which rows need a human.

The same statistics apply to any set of coders, human or not, and an AI assistant can compute them from the coded files, as Prompt 6 below shows. For stage one, where the files are large, the lab ran the same calculation as a script; its output is reproduced here.

Section 2Stage one: three AI coders

Step 4 produced one coding of the 946-article corpus. Stage one adds two more. Prompt 4 is run twice again over the same 19 batches, by two further sub-agents or fresh conversations that receive the same codebook and the same articles and never see the first coder's output or each other's. The result is three independent codings of the same articles, by Coder 1 (the Step 4 dataset), Coder 2 and Coder 3. Comparing them answers a narrow question: given this codebook and this prompt, does the model reach the same codes each time? If it does not, the codebook is under-specified at the points of disagreement, and no human sample is needed to find them. If it does, the codebook is at least consistently applicable, which is necessary for validity but not the same thing.

Before the statistics, the three datasets have to be aligned. Each coder numbered its own rows, so rows are matched on doc_id plus enterprise name, and relevance-0 rows on doc_id alone; the aligned sheets go into dataset_ai_three_coders.xlsx, one sheet per coder plus an unmatched_rows sheet (script: assemble_dataset.py). The coders agreed on the unit of analysis for 1,588 instances. Another 189 were produced by only one or two coders: typically a subordinate mine or workshop inside a 련합기업소 that one coder gave its own row and another folded into the parent, or a farm, fishery or construction site that one coder listed as a relevance-0 row and another left out. These are unit-of-analysis disagreements. They are excluded from the κ calculations, which can only compare rows that exist in every file, and reported separately. They are the most important finding of the stage, because they are the ones the statistics cannot see.

Sixty-five coded values carry a disagreement. Thirty-four are event_type: technical-renovation reports that one coder called construction and another production, rallies at plants coded production by some coders and political_ceremonial by others, completion ceremonies read as political or as construction. Twenty-three are province: two coders inferred a province from the enterprise's name where the third wrote unknown, or placed a plant in a different province. The rest are six leader_visit and two sector values. Every one of them is a boundary that Step 4's ambiguity reports had already named.

Section 3Prompt 5: the human sample

How many rows do humans need to code? For a real project, 10 to 20 percent of the dataset and at least 50 rows, stratified by the coder's confidence so that both the easy and the hard rows are tested. For the workshop, ten rows: enough to run the script and see the statistics behave, little enough that everyone in the room finishes. Ten rows cannot validate anything, and the page says so where it matters. The prompt also asks for a not-relevant row, so that the relevance screen is tested rather than assumed.

# Prompt 5 — Draw the human validation sample

## Context
The AI-coded dataset of the full 946-article corpus is `dataset_ai_coded.xlsx` (sheet `instances`). We now need a sample of rows for human coders to code blind, so that human–AI and human–human agreement can be measured.

## Task
1. Draw a validation sample of **N rows** (workshop default 10, so that every participant can code the whole sample in twenty minutes; for a real project 10–20% of the dataset, and at least 50 rows). Stratify by `coder_confidence` so that both high- and lower-confidence rows are represented (about 60% confidence 3, 30% confidence 2 or 1), and include one or two `relevance = 0` rows so that the relevance screen is tested too. Use a fixed random seed and record it.
2. Write **`human_sample_blank.xlsx`**: the sampled rows with `instance_id`, `doc_id`, `article_date`, `article_long_title`, `article_url` and `article_cleaned_body` kept, and every coded field (`relevance`, `enterprise_name`, `sector`, `event_type`, `leader_visit`, `province`, `evidence_text`, `coder_confidence`, `notes`) emptied. Add a `coder_name` column first. Add a README sheet with coding instructions and a pointer to the codebook. Add Excel data-validation dropdowns from the codebook's `Dropdown_Values` sheet if you can.
3. Write **`human_sample_key.xlsx`**: the same rows with the AI codes retained. This is the instructor's key and is not distributed to coders.
4. Report the sample's composition (confidence strata, years, sectors) against the full dataset so I can see it is not skewed.

## Output
The two workbooks plus a five-line summary: N, seed, strata counts, and anything about the sample you would change.

The workshop sample has 10 rows drawn from Coder 1's dataset with seed 7: six coded at confidence 3, three at confidence 2 or 1, and one relevance-0 row. The blank workbook has every coded field emptied and a coder_name column added; the key, with the AI's codes, stays with the instructor.

Section 4Coding the sample

1. 1Download the blank sample and the codebookBoth files are below. Open the codebook first and read the Variables, category and Decision_Rules sheets once, start to finish. Note the README sheet: it records the interventions the researcher made in Step 3, so you know whose categories you are applying.
2. 2Code all ten rowsFill coder_name, then for each row read the article body (last column), decide relevance, and fill the coded fields from the dropdowns. Copy an evidence excerpt and give your own confidence score. Do not consult the AI or your neighbour; the point is independence.
3. 3Save as human_coded_<yourname>.xlsxKeep the sheet name and column order unchanged; the validation script matches rows on instance_id.
4. 4Hand it inIn the workshop, the instructor collects the files and runs Prompt 6 on the first three completed samples received. Working alone, keep your coders' files in the project folder beside the AI-coded dataset; Section 5 compares them.

• human_sample_blank.xlsx10 rows with coded fields emptied, coder_name column, README with instructions. Code this.13 KB · xlsx
• codebook_nk_enterprises_v2.xlsxThe codebook from Step 3, version 2.0. Keep it open while coding.15 KB · xlsx
• dataset_ai_coded.xlsxCoder 1's dataset, which the human samples are compared against (Step 4).670 KB · xlsx
• dataset_ai_three_coders.xlsxCoders 1, 2 and 3 aligned on instance_id, plus an unmatched_rows sheet, for stage one.815 KB · xlsx

Section 5Prompt 6: stage two, the human validation

Once the human samples are in, the comparison itself is a job for the AI assistant. Open a fresh conversation, upload dataset_ai_coded.xlsx together with every human_coded_<name>.xlsx file, and paste Prompt 6. The prompt asks for the full set of inter-coder reliability statistics: percentage agreement and Cohen's κ for every pair of coders, Fleiss' κ and Krippendorff's α across all coders, the mean human–AI κ and its band, an agreement matrix for each variable, the list of rows on which any two coders disagree, and the verdict under the workshop rule. The matrices are the part worth reading closely: a κ tells you how much disagreement there is, and the matrix tells you where it is, which categories are being confused with which.

# Prompt 6 — Compute inter-coder reliability

## Context
Attached are two kinds of file. `dataset_ai_coded.xlsx` (sheet `instances`) is the AI-coded dataset from Step 4, produced under codebook version 2.0. The files named `human_coded_<name>.xlsx` are blind human codings of a sample of rows drawn from that dataset; each has the same columns as the dataset and a `coder_name` column. Rows are matched across files on `instance_id`. The coded variables are `relevance`, `sector`, `event_type`, `leader_visit` and `province`; `relevance` is compared on every matched row, and the other four only on rows that every coder marked `relevance = 1`, because a non-relevant row carries no codes.

## Task
1. Load every file and match rows on `instance_id`. Report how many rows are matched across all coders and list any `instance_id` that is missing from a file or duplicated within one.
2. For each coded variable, compute:
   - percentage agreement and **Cohen's κ** for every pair of coders (the AI with each human, and each human with each other human);
   - **Fleiss' κ** and **Krippendorff's α** (nominal) across all coders together;
   - the mean of the AI–human κ values, and the band that mean falls in (≥ 0.80 reliable; 0.67–0.80 tentative; < 0.67 revise).
3. For each coded variable, produce the **agreement matrix** (confusion matrix) between the AI and the human coders: categories of the AI coding in rows, categories of the human coding in columns, cell = number of rows, with the pooled humans in one matrix and each human coder in a separate matrix. Mark the diagonal.
4. List every row on which any two coders disagree, with `instance_id`, the variable, and each coder's code side by side.
5. State the overall verdict under the workshop rule: **validated** if the mean AI–human κ of every coded variable is at or above 0.67, otherwise **iterate**, naming the variable(s) that fail.

Compute, do not estimate: use the standard formulas (or a statistics library) and show the n for every statistic. Where a statistic is undefined (for instance a variable with a single category in the sample), say so rather than reporting a number. Do not change any code in any file.

## Output
A report with: (a) the matching summary; (b) one table per variable with every pairwise κ, Fleiss' κ, Krippendorff's α, mean AI–human κ and band; (c) the agreement matrices; (d) the disagreement list; (e) the verdict and two sentences on what the pattern of disagreement suggests about the codebook. Also write the tables to `icr_report.xlsx`, one sheet per section.

Two cautions. Ask the assistant to compute, not to estimate: a capable assistant will run the formulas in code and show the n behind every figure, and you should check that it did, by recomputing one percentage agreement by hand. And keep the files it writes: the report and the icr_report.xlsx workbook are the validation record for this codebook version and belong in the project folder beside the run log.

How to read the result. Look first at the human–human pairs: they are the ceiling. If humans agree with each other at κ = 0.4 on a variable, no AI can be expected to do better, and the problem is the category definitions. Next compare each AI–human κ with that ceiling: an AI that agrees with each human about as well as they agree with each other is doing the job. Then read the band for the whole variable. Finally set the agreement matrices beside the stage-one disagreements: a variable on which the AI coders agreed with one another but not with the humans is being coded consistently and wrongly, and that is a codebook definition to rewrite, not a rule to add.

Section 6Accept or iterate

The decision rule the workshop uses: if the mean human–AI κ is at or above 0.67 for every coded variable, the dataset is validated for that codebook version and may be analysed. If any variable falls below, iterate: return to Step 2 for the variable in question (what does the corpus actually say?), revise its typology and rules in Step 3 as codebook version 2.1, extract again in Step 4, and validate again. Two or three iterations are normal. Every iteration is logged with its version number and both stages' κ values, so that the final paper can report how the instrument improved.

For this dataset, stage one is passed and stage two is pending the workshop. The agenda for version 2.1 is already written: the 189 unit-of-analysis mismatches and the 65 disagreements above, together with the ambiguity reports in Step 4, point at the same four places in the codebook (subordinate mines, 기술개건, rallies at plants, and province inference).

Two refinements for real projects. Set the threshold by variable: the variable that carries the argument may need α ≥ 0.80 while a descriptive control can live with 0.67. And use the confidence score: if high-confidence rows validate and medium-confidence rows do not, scale the extraction, send the medium-confidence rows to human review, and report both figures.