Step 4 · Dataset Extraction

Extraction is the step people imagine when they think of "AI coding", and it is the least interesting one. By now the model has no decisions to make about the design. It reads each article, applies the codebook, and writes rows. What matters is that every row carries the evidence for itself, and that the model is told to report the places where the codebook let it down.

Section 1From codebook to extraction

The workshop codes the full dataset: all 946 articles of the corpus assembled in Step 1, with no pilot sample. A corpus of this size is small enough to code in full, and coding everything means the analysis describes the corpus itself rather than an estimate from a sample. The articles are split into 19 batches of about 50, in corpus order, so that each batch fits comfortably in one conversation and a failed batch can be rerun on its own. Each batch is a JSON file with each article's identifier, date, category, title and cleaned body.

This page follows a single coder: one coding agent, given codebook version 2.0, coding every batch and producing one dataset. That is the whole of the extraction step. Whether the dataset can be trusted is a separate question, and it is answered in Step 5, where the same corpus is coded again by two further independent coders and the codings are compared before any human reads a row. With a much larger corpus, run this step on a stratified sample first and scale up only after Step 5 passes.

Section 2Prompt 4: structured extraction

The prompt gives the model the codebook, one batch of articles, and a table of field rules. Four constraints do most of the work: no invented enterprises, visits or locations; the event-precedence rule applies only when events receive comparable attention; quotations of the leader are not visits; and the evidence excerpt must be an exact substring of the article body, which the model is told to check before it finishes. The reply asked for is deliberately short, so that the model spends its output on the data file rather than on narration.

# Prompt 4 — Structured extraction with the codebook

## Context
You are one of several independent AI coders for a dataset extraction. Work only from the codebook and the articles given here. The codebook is `codebook_nk_enterprises_v2.xlsx` (also `codebook.json`). The articles to code are in `batch_<b>.json` (the 946-article corpus from Step 1, split into 19 batches of about 50 articles): a list of objects with `doc_id`, `article_date`, `article_category`, `article_long_title`, `article_cleaned_body`. Read the codebook completely before coding, including Decision_Rules.

## Task
Code every article. Produce one row per **instance** (one named enterprise in one article). Where the article names no specific enterprise, produce one row with `relevance = 0` and all other coded fields empty.

For every row supply exactly these fields:

| field | rule |
|---|---|
| `doc_id` | copy from input |
| `instance_id` | `<doc_id>-<n>`, numbering enterprises in order of first appearance |
| `relevance` | 1 or 0 per the codebook |
| `enterprise_name` | the name exactly as written in the article |
| `sector`, `event_type`, `leader_visit`, `province` | one allowed code each; no new codes, no blanks on relevant rows |
| `evidence_text` | a verbatim excerpt (one or two sentences) copied from `article_cleaned_body` that justifies sector and event_type |
| `coder_confidence` | 3, 2 or 1 per the codebook |
| `notes` | optional: the boundary case you faced and the rule you applied |

Constraints:
- Never invent an enterprise, a visit or a location that the text does not support. If province cannot be determined from the text or the enterprise's name, code `unknown` and lower confidence.
- Apply the event precedence order in Decision_Rules only when two events receive comparable attention.
- Quotations of the leader's instructions are not a visit.
- Check at the end that every `evidence_text` is an exact substring of the article body and that every `doc_id` in the input appears at least once.

## Output
Write the rows to `coded_batch_<b>.json` (a JSON array); the batches are assembled afterwards into `dataset_ai_coded.xlsx` with the Coding_Template columns. Code every batch the same way, in a fresh conversation, without seeing any other coder's output. Reply with only: the row count, the number of relevance-0 rows, and a bullet list of codebook ambiguities you met (doc_ids and the rule you wished existed). These ambiguities feed the next codebook revision.

Section 3Running the extraction

1. 1Split the corpus into batchesIn your coding agent: "Split the 946-article corpus into 19 batches of about 50 articles in corpus order, save them as extraction/batch_1.json to batch_19.json, and record the batch boundaries in extraction/run_log.md."
2. 2Run Prompt 4 on every batchOne coding agent, given the codebook and one batch, writes extraction/coded_batch_<b>.json and returns a short reply: the row count, the number of relevance-0 rows and a list of codebook ambiguities. Start a fresh conversation for each batch, and log each reply in the run log. The workshop run returned 1,702 rows across the 19 batches.
3. 3Check the output mechanicallyA short script, or the agent itself: every code is in the codebook's allowed list; every doc_id in the corpus appears at least once; every evidence_text is a substring of its article's body. The workshop run passed all three checks: zero invalid codes, no missing articles, and every excerpt found verbatim.
4. 4Assemble the datasetMerge the coded rows with the article metadata (date, year, category, title, URL) into dataset_ai_coded.xlsx, the dataset used in Step 5 and on the Analysis page, with the article body in the last column so that human coders in Step 5 can read it without a second file. Script: assemble_dataset.py.
5. 5Collect the ambiguity reportPaste the bullet list the coder returned into codebook/ambiguities_v2.md. It is the agenda for codebook version 2.1.

Section 4The full dataset

The assembled dataset has 1,702 rows from all 946 articles. 126 rows are relevance 0: articles that name no enterprise, or whose enterprise is a farm, fishery or other unit the scope rule puts outside the study. The remaining 1,576 rows are relevant instances from 861 articles, an average of 1.8 enterprises per relevant article, because a leadership tour or a mobilisation report often names several plants. Five sample rows:

• dataset_ai_coded.xlsxSheet instances: 1,702 rows × 18 columns including evidence_text, coder_confidence, notes and the article body. Sheet README.670 KB · xlsx

Section 5What the coder reported back

Across the 19 batches, the coder's replies listed the places where codebook 2.0 did not decide the case, and the same problems recurred batch after batch. Read together they are a precise description of the codebook's weaknesses, and they are the reason the extraction prompt asks for them. Each becomes a candidate rule or category in version 2.1, to be settled by the researcher, not the model. Consolidated:

• Province. The most frequent note by far. Most articles do not state a location; coders inferred it from the enterprise's name and lowered confidence, or coded unknown. Some names recur dozens of times without a listed location (남흥, 상원, 대안, 룡성, 천리마, 검덕), and 황해제철 could be North or South Hwanghae. If province matters to the analysis, a gazetteer of enterprise locations belongs in the codebook.
• Subordinate mines inside a 련합기업소. When an article names a combine and two of its mines, is that one instance or three? The coder asked, and in Step 5 this turns out to be the main source of unit-of-analysis mismatches between independent coders.
• Out-of-scope units alongside in-scope ones. A leadership tour that visits a steel works and a cooperative farm yields one relevant row and one note; the coder asked whether the farm should get a relevance-0 row of its own. Rule 1 says no, but the prompt should say so explicitly.
• Sector edge cases. 2.8비날론련합기업소 (chemical fibre: heavy or light?), 고려약공장 (pharmaceuticals), glass, rubber, wood-processing and ceramics plants, 제염소 (salt: mining by rule, but reads as light industry), and repair, mining-machinery and instrument plants. All were coded heavy_industry under rule 3 except salt; the codebook should list them.
• Event boundaries. Agitation and propaganda work at a plant (political_ceremonial or production?); 기술개건 or a new process line as construction_modernization versus production; blasting at a mine; flood-prevention work; award ceremonies, thank-you letters and congratulatory telegrams; anniversaries of a past leadership visit.
• Which officials count as present. A provincial Party secretary at a rally, or a minister sending a delegation, tested the premier_cabinet boundary. Rule 6 covers attendance but not rank.
• Venue-only mentions. Art-troupe performances and sports events held at an enterprise: the enterprise is named but nothing is reported about it. Coded political_ceremonial with low confidence; a venue rule is needed.
• Identifiable but unnamed enterprises (중요국방공업기업소) and a relayed foreign-press report about a North Korean plant were coded relevance 0 under rule 7 and the foreign-enterprise clause; both are worth a sentence in the methods.

None of these are reasons to distrust the dataset yet. They are reasons to measure it, which is Step 5. Note that several of them are consequences of the researcher's interventions in Step 3: collapsing sectors created the chemical-fibre and salt questions, and the scope rule created the farm-beside-a-steel-works question. Interventions have costs, and the ambiguity reports are where they show up.