Organization of Power

Generating Data I: Extracting Datasets from a Large Media Corpus

Overview

Video not available yet

Estimated duration: 120 minutes

Overview

A two-hour, hands-on tutorial: turn a large text corpus into a validated dataset with an AI assistant, worked through on North Korean newspaper articles.

Scope

A hands-on tutorial on using large language models to build a validated research dataset from a large text corpus, in five supervised steps, followed by a first analysis.

Format

Tutorial · 6 pages · about 2 hours

Worked example

Industrial enterprises in Rodong Sinmun and KCNA, from the lab's nkmedia database

Social scientists increasingly have access to corpora of hundreds of thousands of documents: news archives, parliamentary records, court filings, leaked document sets, social-media exports. Reading them is impossible; keyword counts throw away most of what they contain. Large language models (LLMs) can read them for us, but an LLM left to itself will quietly decide what the research question is. This tutorial shows how to keep those decisions in human hands while letting the model do the reading, the proposing, the structured coding and much of the quality control.

The workflow has five steps. You narrow a corpus with a keyword search (Step 1), ask the model to explore it open-endedly (Step 2), turn what it finds into a codebook and a machine-readable schema that you have approved (Step 3), use that codebook to extract a dataset with one row per unit of analysis (Step 4), and validate the extraction against human coders with standard inter-coder reliability statistics (Step 5). Steps 2 to 5 are a loop: if reliability is too low you revise the codebook and extract again. Only a validated dataset goes on to analysis.

Every page uses one running example: articles about industrial enterprises (기업소) in the North Korean newspapers Rodong Sinmun and the Korean Central News Agency, drawn from the lab's nkmedia database. The files produced at each step are downloadable, so you can join the workflow at any step, or substitute a corpus of your own.

Figure 1The five-step workflow and the validation loop
Five-step workflow Keyword search, open-ended exploration, framework and codebook, dataset extraction, and validation lead to analysis. A loop returns from validation to exploration until the dataset is validated. HUMAN DECIDES · AI READS, PROPOSES, CODES STEP 1 Keywordsearch 310,063 → 946 articles STEP 2 Open-endedexploration topics · entities · events STEP 3 Framework &codebook typologies · schema STEP 4 Datasetextraction 1 row per instance STEP 5 Validation AI × human coders κ, Fleiss κ, Krippendorff α THEN Analysis univariate → multivariate if reliability is too low: revise the codebook and extract again Human: research question · unit of analysis · variables · codebook approval · validation threshold · interpretation
The model never fixes the research design. At each step a human reviews what it proposes, and the codebook it must follow in Step 4 is the one the researcher approved in Step 3. Figure adapted from the presentation slides.

What you need

  • A laptop with a spreadsheet program. Python is helpful for the Analysis page but is not required: every file is also provided ready-made.
  • Access to a capable AI assistant. A web interface with file-upload and projects (Claude, ChatGPT, Gemini) is enough. A desktop coding agent (Claude Code, Codex) is better; Step 2 explains both.
  • The workshop files. Each step page links the files it needs. The full set is listed in Step 1, Section 5.
  • Optionally, your own corpus: a CSV or XLSX of documents, or a ZIP of PDF, DOCX, TXT or HTML files, if you are authorised to share it with an AI service and it contains nothing sensitive. The prompts on each page say what to change.

How to read this

Prompts are shown in full, in boxes you can copy. They follow one pattern, Context · Task · Output, so that the model knows what it is looking at, what to do and what to hand back. Tables and figures are the actual output of the workshop run, not illustrations; where a result is simulated the page says so. Items marked TBD are still being decided for the published version of this tutorial.

The presentation slides and the accompanying paper give the method's rationale and a full-scale application, including validation results on a 1.6-million-article corpus. This tutorial is the small-scale, do-it-yourself version.

Content

Work through the overview and the five steps in order; the Analysis page shows what the full dataset supports. Every step has downloadable files and the exact prompts used, so the whole workflow can be reproduced with a spreadsheet and a capable AI assistant. Prepared for the Ewha Womans University workshop “AI and Machine Learning in Social Science” on 30 October 2026.

Steps 1–5 From corpus to validated dataset Each step is one page. Work through them in order the first time.
  1. 1Keyword searchSize up the nkmedia corpus, choose a keyword that defines the subject, restrict the corpus to 946 articles and set up a local project folder.
  2. 2Open-ended explorationHand the corpus to an AI assistant, in a web project or a desktop coding agent, and ask it to describe topics, entities, events and relationships before any variable is fixed.
  3. 3Framework and codebookTurn the exploration into a research design: a unit of analysis, four variables, a typology for each, revised by the researcher, and a codebook workbook the model and your coders will both follow.
  4. 4Dataset extractionRun a structured extraction prompt over all 946 articles from Step 1, batch by batch, and inspect the resulting dataset, with its evidence excerpts and confidence scores.
  5. 5ValidationCompare three independent AI codings of the full dataset, then draw a sample for human coders and have an AI assistant compute Cohen's κ, Fleiss' κ, Krippendorff's α and an agreement matrix for every variable. Decide whether to accept or iterate.
Then Analysis What a validated dataset lets you say.
  1. 6AnalysisUnivariate distributions, cross-tabulations by period and sector, and a logistic regression of leadership visits on the full dataset, with the caveats the validation results impose.

Workshops

Resources

Relevant Literature

Browse references and supporting literature for this training module in the Organization of Power Lab Zotero library.

Open this training module's Zotero collection

Corpus: nkmedia v1 (KCNA and Rodong Sinmun, 1998–2025), Organization of Power Lab. Tutorial prepared for the workshop AI and Machine Learning in Social Science: Promises and Potentials for Future Research, Ewha Womans University, 30 October 2026, supported by the National Research Foundation of Korea, National Chengchi University and the Ewha Institute of Politics. Program details are tentative until confirmed by the organisers.