Org Power Lab · National Chengchi University

AI-Human Workflow for Extracting and Validating Datasets from Large Media Corpuses

Jacob Reidhead · National Chengchi University

AI and Machine Learning in Social Science: Promises and Potentials for Future Research
Panel 2 · LLM in Social Science Research: Methods & Applications
Ewha Womans University, Seoul · Friday, October 30, 2026

1 / 23

Org Power Lab

How long would it take to read?

1,655,741articles in the Kazakhstan Media Database

13,640candidate articles on Central Asian cooperation

1research question: who coordinates regional cooperation, and did that change after February 2022?

Reading the 13,640 candidates once, at 400 words per article and 250 words per minute

One research assistant

≈ 9 working weeks

A team of five

≈ 2 working weeks

An LLM, in batches

hours

Rough estimate for reading only, before any coding. The full database would take one reader about twenty years.

Reidhead · AI-Human Workflow · Ewha, October 30, 20262 / 23

Org Power Lab

The claim: an LLM coder is a research assistant, not an instrument

Needs detailed instruction

a codebook, not a question

Improves with iterative training

pilot, inspect, revise

Output is non-deterministic

same article, two occasions, sometimes two codes

Treat it like any coder

double-check a sample · spot-check the rest · multiple coders · intercoder reliability

Reidhead · AI-Human Workflow · Ewha, October 30, 20263 / 23

Org Power Lab

What the assistant should not decide

The LLM accelerates

Reading

Discovery

Coding at scale

Quality control

The researcher decides

Unit of analysis

Variables

Evidence rules

Inferences

Reidhead · AI-Human Workflow · Ewha, October 30, 20264 / 23

Org Power Lab

A five-step workflow, then analysis

Click a step

Researcher

LLM

Output

Reidhead · AI-Human Workflow · Ewha, October 30, 20265 / 23

Org Power Lab

Who does what at each step

Step	Researcher decides	LLM contributes	Output	AI gain
1 Prepare corpus	Scope, outlets, languages, years, search terms	Can write scrapers, clean, deduplicate, identify language, expand keyword forms. Here: keyword forms only, corpus pre-existed	Candidate corpus with stable IDs	varies
2 Explore	Which dimensions to explore	Reads the whole candidate corpus; lists, typologizes, counts, clusters, traces over time	Corpus inventory, exploratory typology	transforms
3 Framework	Unit, variables, categories, decision and evidence rules	Proposes typologies, stress-tests definitions, drafts codebook text	Codebook and machine-readable schema	assists
4 Extract	Model, settings, sample, multiple instances per document	Applies the codebook to every document; records evidence and confidence	Instance-level dataset with evidence and confidence	transforms
5 Validate	Sampling design, reliability thresholds, what to do with unreliable categories	One of several coders; re-codes after revision; summarizes disagreements	Reliability table, revised codebook, error log, frozen dataset	assists

Table 1 of the paper. The researcher has the final word at Step 3; the model has the largest effect at Steps 2 and 4.

Reidhead · AI-Human Workflow · Ewha, October 30, 20266 / 23

Org Power Lab

The case: who coordinates regional cooperation in Central Asia?

Joint project with Anar Shaikenova and Nygmet Ibadildin, KIMEP University, Almaty

Two decades of "failed or externally imposed" integration, then a revival of Central Asian summitry since 2018. Did the region decouple from Russia after February 2022, and if so, toward what?

Three regionalisms, by coordinating actor

Russia-led: EAEU, CSTO, CIS
Large-power-led: China, US, EU, Türkiye, Gulf states
Indigenous: Central Asian governments themselves

Hypotheses after February 2022

H1 Russia-led cooperation lost salience qualified
H2 Large-power-led cooperation gained supported
H3 Indigenous cooperation gained not supported (share)
H4 Shifts differ by sector supported

Needs a dataset of cooperation instances coded by coordinator and sector over time. That is what the workflow produced.

Reidhead · AI-Human Workflow · Ewha, October 30, 20267 / 23

Org Power Lab

Step 1: from 1.66 million articles to 13,640 candidates

Articles

1,655,741

Kazakhstan Media Database v3.0

Sources

11 outlets + presidential site (Akorda)

Languages

74/22/3

% Russian / Kazakh / English

Articles by year of publication. *2026 to June. Archive depth grows over time, so the analysis normalizes by candidate articles per year.

Nine Boolean queries, Russian and Kazakh, inflected forms

Every query: a Central Asia term (24 forms) + a cooperation term (69 forms)
Six sector queries (security, water, energy, trade, production, people): 13,972 articles
Three coordination queries (Russia-led, large-power-led, indigenous = none of 398 external-actor terms): 14,355 articles

Intersection13,640candidate articles

Query labels only select candidates. Coordinator and sector are coded from the text.

Reidhead · AI-Human Workflow · Ewha, October 30, 20268 / 23

Org Power Lab

Step 2 · Exploration: the model reads the whole candidate corpus

Step 2 Prompt · Open-ended analysis · ChatGPT 5.6 Sol, high reasoning, May 2026

Researcher

## Context
"corpus.xlsx" - a dataset of articles from KZ media, selected by
querying references to Eurasian integration

## Open-Ended Analysis
1. EXTRACTION_DIMENSION: themes, actors, countries, institutions
   and organizations, events, relations
2. Extract and read the document_text field of "corpus.xlsx"
3. For each EXTRACTION_DIMENSION, generate a list of relevant items,
   a typology, add a brief summary, list the article id ...
4. Write a general overview.
5. Including a summary of corpus statistics.
6. Display results as text and also export ... "1 Open-ended Analysis.pdf"

Run on an earlier candidate corpus of 6,962 articles (five outlets, 2021–2025). A second prompt asked for clusters and change over time.

Assistant

A 1,035-line, auditable inventory

Themes by article count: regional diplomacy and summitry 4,916; agriculture 3,065; trade and customs 3,017; finance and investment 2,838; security 2,642; industrial policy 2,559 …
Actors: heads of state and governments dominate, not experts or publics
Over time (2021→2025): Russia's theme share down about 14 points, security down 13; industry, trade, energy, China, digitalization up
Every item carries the article IDs that support it

We treated these numbers as hypotheses, not findings. They became H1–H4.

Reidhead · AI-Human Workflow · Ewha, October 30, 20269 / 23

Org Power Lab

Step 3 · Training the coder: expand the codebook from the pilot

Step 3 Prompt · Expand codebook definitions from pilot coding

Researcher

# Context
## CodeBook_CA integration Project_1.docx
## [coder1].xlsx, [coder2].xlsx - human coded articles on CAI

# FIVE_VARIABLES: Relevance, Coordination_dimension,
  Sector_dimension, Bilateral_ca_cooperation, coder_confidence

# Open [coder1].xlsx, [coder2].xlsx. For each of the FIVE_VARIABLES,
  group articles coded by each category
# Use these article groups ... to expand the definition of each
  variable category in the codebook.
# Create a copy of the codebook, CodeBook_CA integration Project_2.docx.
  Preserve the current structure ... update the definitions.

The single prompt we would most recommend. Grounding the definitions in the coders' own decisions is what "training" an LLM coder consists of.

Assistant

Codebook draft v2 · values proposed for the five variables

Variable	Values proposed
Relevance	1 relevant · 0 not relevant
Coordination	Russia-led: EAEU; CSTO; CIS; Russia bilateral · Large-power-led: China (BRI, China–CA); United States (C5+1); EU (Global Gateway); Türkiye (OTS); Gulf states; Japan; South Korea; India; Iran · International: World Bank; ADB; EBRD; AIIB; UN agencies · Indigenous: Consultative Meetings of CA heads of state; CA bilateral; CA governments' own programs
Sector	Security; Defense; Border; Water; Environment; Climate; Energy; Oil and gas; Minerals; Electricity; Trade; Transport and corridors; Customs and logistics; Industry; Technology and digital; ~~Finance and investment~~; Agriculture; Education and science; Culture; Tourism; Migration and labor; Health; Multidimensional
Bilateral CA	1 yes · 0 no
Confidence	3 high · 2 medium · 1 low

Every value with a definition, "code when" and "do not code when" rules drawn from the articles the coders had placed there (Figure 2). Far more values than the codebook kept: next slide.

Reidhead · AI-Human Workflow · Ewha, October 30, 202610 / 23

Org Power Lab

Step 3 · What the model proposed and what the researchers kept

Model's draft values (codebook v2 draft)

Coordination · 21 values

EAEUCSTOCISRussia bilateralChina / BRIUnited States (C5+1)EU (Global Gateway)Türkiye (OTS)Gulf statesJapanSouth KoreaIndiaIranWorld BankADBEBRDAIIBUN agenciesConsultative MeetingsCA bilateralCA governments' programs

Sector · 23 values

SecurityDefenseBorderWaterEnvironmentClimateEnergyOil and gasMineralsElectricityTradeTransport and corridorsCustoms and logisticsIndustryTechnology and digitalFinance and investmentAgricultureEducation and scienceCultureTourismMigration and laborHealthMultidimensional

Relevance · Bilateral CA · Confidence

1 / 01 / 03 / 2 / 1

Researchers' selection (Table 5)

Coordination · 3 values

Russia-ledLarge-power-ledIndigenous

The regionalism debate is about three contenders, not twenty-one actors. Development banks and UN agencies, companies by home country, and Central Asian governments' own programs are folded in by decision rule.

Sector · 7 values

SecurityWater / Environment / ClimateEnergy / Oil / Gas / MineralsTrade / Transport / LogisticsProduction / Industry / Technology / DigitalPeopleMultidimensional

Fine-grained fields merged into the policy areas the literature compares. Finance dropped after the pilot; "People" added to hold education, culture, tourism and migration.

Relevance · Bilateral CA · Confidence

1 / 01 / 03 / 2 / 1

Reidhead · AI-Human Workflow · Ewha, October 30, 202611 / 23

Org Power Lab

Step 3 · The codebook adopted (Table 5)

Variable	Values	Rule
Relevance	1 / 0	Cooperation, coordination, partnership or integration among two or more Central Asian states or the region, whether or not the word "integration" appears
Coordination	Russia-led (EAEU, CSTO, CIS) · Large-power-led (China, US, EU, Türkiye, Gulf formats) · Indigenous (Central Asian governments)	Who primarily leads or facilitates. A mention of an external power does not by itself make an instance externally led. IFI- and UN-facilitated projects, company cooperation by home country, and CA governments' own programs: indigenous
Sector	Security · Water/Environment/Climate · Energy/Oil/Gas/Minerals · Trade/Transport/Logistics · Production/Industry/Technology/Digital · People · Multidimensional (residual)	Primary policy field, decided in order by headline, lead paragraph, main purpose, and bulk of discussion. Multidimensional when three or more sectors get comparable attention
Bilateral CA	1 / 0	Instance primarily concerns two Central Asian states (indigenous only)
Confidence	3 / 2 / 1	Coder's confidence in the decision; carried into the dataset and used to stratify validation

Table 5, coding scheme after framework revision. The model's draft values were collapsed to what the regionalism literature can use: three coordinators and six sectors plus a residual. "Finance and investment" was dropped and "People" added after the pilot rounds (slide 16). The model's own earlier proposal, a 3×3 actor-by-action design, was set aside because the question is who coordinates, not who speaks.

Reidhead · AI-Human Workflow · Ewha, October 30, 202612 / 23

Org Power Lab

Step 4 · Extraction: 1,743 instances from the full candidate corpus

Step 4 Prompt · Final extraction run, 9 July 2026 (essentials)

Researcher

# Context
## CodeBook_CA integration Project_2.xlsx - read the Variables,
   Sector_Codes and Coordination_Codes sheets; ignore the
   keyword dictionary
## corpus_2.0.xlsx - candidate articles, database v3.0

# Extract every instance of CA regional integration in one of
  SIX_SECTORS, coordinated by one of THREE_COORDINATORS.
# Code the FIVE_VARIABLES; record evidence text and confidence.
# Different sectors or coordinators in one article -> separate
  rows, each with its own Instance_ID.
# Return nothing for an article with no relevant instance.

Practical issues: batched uploads, early stops, targeted manual corrections logged in the output's README.

Assistant

Instances

1,743

from 1,716 articles, 2009–2026

High confidence

70.6%

1,231 instances; 29.4% medium; none low

Confidence is not uniform across categories

Coordinator	Instances	%	High conf.
Indigenous	1,118	64.1	59.4%
Large-power-led	436	25.0	89.7%
Russia-led	189	10.8	93.1%

Russia-led cooperation is signaled by institutional names; indigenous cooperation is a residual. Dropping medium-confidence rows removes indigenous instances disproportionately.

Reidhead · AI-Human Workflow · Ewha, October 30, 202613 / 23

Org Power Lab

Step 5 · Validation protocol: one AI coder, two human coders

Three coders, each coding independently on their own sheet

Claude

AI coder · codes the full corpus; a 500-instance stratified sample of its output is what the humans check

Coder 1

Human · codes the 500-instance sample, blind to the AI's codes

Coder 2

Human · codes the same 500 instances, blind to the AI's codes and to Coder 1

Same codebook, same unit of analysis, same evidence rules. Human coders: Alima and Alina, undergraduate research assistants at KIMEP University. Sample: 300 high- and 200 medium-confidence instances, drawn 28 July 2026.

Inter-coder reliability · Round 3, high-confidence stratum (Table 7)

	Claude	Coder 1	Coder 2
Claude	—
Coder 1		—
Coder 2			—

Pairwise Cohen's κ (large) and % agreement (small): coordination.

Krippendorff's α

three coders, high-confidence

Instances

compared by all three coders

Relevance confirmed

92.0%

of the AI's high-confidence instances

Reidhead · AI-Human Workflow · Ewha, October 30, 202614 / 23

Org Power Lab

Reliability: the model's confidence rating tells you which codes to trust

High-confidence stratum (n = 276 coordination, 164 sector) Medium-confidence stratum (n = 177, 60). Validation sample, 28 July 2026 (Table 7, Figure 5). The matrix on the previous slide is the high-confidence stratum.

Three results

High confidence: the LLM agrees with each human about as well as the humans agree with each other. Coordination κ 0.80 / 0.68 vs 0.81; sector 0.86 / 0.81 vs 0.88. Krippendorff's α 0.77 and 0.85.
Medium confidence: humans still agree (κ 0.88) but human–LLM agreement on coordination falls to 0.49 / 0.58. The model was telling us, accurately, which of its own codes not to trust.
Humans confirmed relevance of 92.0% of the high-confidence sample, bounding the false-positive rate.

Decision: analyze high-confidence instances only. Taken on reliability grounds, but not content-neutral.

Reidhead · AI-Human Workflow · Ewha, October 30, 202615 / 23

Org Power Lab

Three rounds of validation, and what reading the disagreements changed

Three rounds

Round 1

12 June 2026

limited corpus, codebook v1; disagreements read, κ not computed

expanded corpus

Round 2

18 June 2026

low agreement on "finance" and the residual categories

altered framework, codebook v2

Round 3

28 July 2026

300 high- and 200 medium-confidence instances; α 0.77 / 0.85

What reading the disagreements changed

Dropped "finance" as a sector

Humans disagreed about it too: finance is a dimension of nearly every field, not a field of its own. Low reliability exposed a flaw in the typology, not in the coders.

Added "people-to-people"

Education, culture, tourism, migration and public diplomacy had no category and were being forced into trade or multidimensional.

Three decision rules in codebook v2

IFI- and UN-facilitated projects are indigenous. Company cooperation is coded by the company's home country. Central Asian governments' own programs are indigenous.

Reidhead · AI-Human Workflow · Ewha, October 30, 202616 / 23

Org Power Lab

From corpus to analytic sample

Step	Instances	Articles	Human- validated
Kazakhstan Media Database v3.0	—	1,655,741	—
Nine Boolean queries → candidate articles	—	13,640	—
Extracted by the LLM (Step 4)	1,743	1,716	500
− 512 medium-confidence instances	1,231	1,205	300
− 24 judged not relevant by humans	1,207	1,185	276
− 7 multidimensional → analytic sample	1,200	1,178	274

926 of the 1,200 analysis instances were never read by a human. Their relevance is inferred from the 92.0% confirmed in the 300-instance sample.

Frozen with a run manifest

Dataset = 9 July 2026 extraction, corpus 2.0 from database v3.0
README sheet generated with the output: time, source workbook and codebook, counts, date range, confidence definitions, limitation statement, marginals of every variable
High- and medium-confidence subsets stored as separate workbooks

Known bias: medium-confidence instances are disproportionately indigenous, so the analysis understates the indigenous share. We report the direction wherever it matters.

Reidhead · AI-Human Workflow · Ewha, October 30, 202617 / 23

Org Power Lab

Findings I · The three regionalisms are not competing for the same ground

Russia-led Large-power-led Indigenous. Analytic sample n = 1,200 (Table 9).

Indigenous coordination is most frequent overall (54%) and dominates water (78%), trade and transport (63%), and people-to-people (63%).
Large powers account for just under a third and are strongest in security (40%) and production and technology (40%).
Russia is a small share overall (14%), concentrated in security, where it is a third of instances.

As reported in Kazakhstan's media, Russia's regional presence is a security presence; the Central Asian states' own cooperation is economic, environmental, and cultural.

Reidhead · AI-Human Workflow · Ewha, October 30, 202618 / 23

Org Power Lab

Findings II · After February 2022 the shift went to other large powers, not to the region itself

Instances per 1,000 candidate articles, 2014–2026 (*to June). Table 10, Figure 6. Click a legend button to hide or show a series.

Multinomial logit, post-Feb 2022 average marginal effects (M2)

P(coordinator)	Δ pp	(SE)
Large-power-led	+14.2***	(2.9)
Indigenous	−11.3***	(3.3)
Russia-led	−2.9	(2.5)

Largest shifts in people-to-people (+24.3 large-power) and trade and transport (+13.7); none in security or water
2022 Russia-led spike = post-invasion summitry (CSTO anniversary, first CA–Russia summit); decline begins 2023

H2 supported; H1 only in qualified form; H3 not supported in share terms; H4 supported.

Reidhead · AI-Human Workflow · Ewha, October 30, 202619 / 23

Org Power Lab

Implications and shortcomings

Four implications

The analogy is a practical guide

write, pilot, revise, measure, decide

Collect model confidence

κ ≈ 0.8 vs κ ≈ 0.5, at no cost

Validation improves the framework

every disagreement resolved in writing

Division of labor

hold the line at the framework step

Four shortcomings

Salience, not cooperation

as reported in Kazakhstan's media

Keyword-selected corpus

indigenous is a residual category

Only high-confidence validated

926 instances rest on a 300-instance sample

A single AI coder

a second model is a cheap check we did not run

Reidhead · AI-Human Workflow · Ewha, October 30, 202620 / 23

Org Power Lab

The tools change; the principles of research do not

Claims require evidence

AI can inventory a corpus and locate the evidence; the claim still rests on it

Data require validation

AI can code at scale, even as a second coder; agreement is still measured

Researchers choose frameworks

AI can propose and refine a typology; the researcher still decides

Reidhead · AI-Human Workflow · Ewha, October 30, 202621 / 23

Org Power Lab

Future steps

Expand

the same workflow on the lab's other corpora: Rodong Sinmun and KCNA, the Epstein Files

Systematize

reusable prompts, schemas and run manifests; a second AI coder as routine, not an experiment

Adapt

bring the literature review into the framework step, so categories arrive with their lineage

Reidhead · AI-Human Workflow · Ewha, October 30, 202622 / 23

Org Power Lab

Org Power Lab · orgpowerlab.org

Unveiling power in politics with organization theory and data science.

Publications

6 published · 8 preprints · 4 in progress

Approaches

Networks & Organizations · Text as Data

Regions

North and South Korea, Taiwan, China, Kazakhstan, global

Current

SSRN

preprints on elite management, party organization, state attention

Ibadildin, Reidhead, Bazarov & Talipova (2025). Liberal framing in illiberal regimes: January 2022 protest violence in Kazakhstan's state-controlled media. Text as Data
Song, Reidhead & Park (2025). Introducing career transition data on elites in North Korea. Journal of East Asian Studies
Reidhead (2026). By firm, not by region: central management and state attention in North Korea. SSRN
Reidhead & Kao (2026). Organizational inheritance and conditional centralization: regional party leadership in Taiwan. SSRN

Politics

Database · Taiwan

Taiwan Politics

50,902 officials · 95,433 candidacies · 2,954 family ties

Database · South Korea

South Korea Politics

40,175 officials · 14,964 candidacies · 12,694 family ties

Database · North Korea

North Korea Politics

1,048 party and government organizations, officials, careers

Media

Database · Kazakhstan

Kazakhstan Media

3.05M news articles, most 2009–2026

Database · North Korea

North Korea Media

310,063 KCNA articles, 1998–2025

Database

Eurasia Media

TBD cross-country news corpus, in preparation

Economics

Database · North Korea

North Korea Enterprises

3,674 enterprises with KCNA references

Database · Kazakhstan

Kazakhstan Critical Minerals

TBD projects, firms and partners, in preparation

Database · South Korea

South Korea Business Elites

TBD executives and board members, in preparation

Professionals

Database · South Korea

South Korea Lawyers

7,884 profiles at 21 firms (uploading)

Cross-field

Database · Global

Epstein Files

2.44M documents, 1990–2025

Tutorials

Python Tutorials

Python data types, control structures, functions, and classes.

Tutorials

AI × Corpus to Dataset

An AI-human workflow for extracting and validating data from large text and media corpora. This afternoon, 3:00–5:00.

Tutorials

AI × Literature Review

Searching, screening, extracting and synthesizing a literature with AI assistants, with human review at every stage.

Courses

Computational Social Science

A survey of text analysis, machine learning, and social network analysis for social-science research.

Courses

Data Engineering and Analysis

The data-science workflow from collection to analysis, with hands-on emphasis on engineering, cleaning, curation, and wrangling.

Courses

Networks & Organizations

A graduate-level introduction to organization theory and its integration with network perspectives.

Tutorials, nine courses and this workshop at orgpowerlab.org/training.

Data scientists

Chih-yi Su

Data Scientist

BA Economics, BS Mathematical Science

Liang-Ta Wen

Data Scientist

BA Money & Banking, BS Mathematical Science

Martin Chen

Data Scientist

BS Computer Science

Archival researchers

Bonggyu Lee

Archival Researcher

PhD Asia-Pacific Studies

Yi-Chieh Chung

Archival Researcher

MA Diplomacy

Emily Kao

Archival Researcher

BA Korean Language and Culture

All at National Chengchi University, Taipei. Student data scientists build and audit the databases; archival researchers collect and code. Collaborators and independent researchers co-author from partner institutions (Collaborate tab).

Share data

every database has a researcher login; ask us

Collaborate

bring a question, expertise, or a textual source

Train

tutorials, courses, workshops; we travel

Graduate Student Collaborators

Chia-Lin KaoPhD, Rutgers University
Mukhamejan BazarovMA, University of Glasgow
Ayazhan YerzhomartovaMA, Tsukuba University
Zhang YangMA, Hong Kong Metropolitan University
Maiya TalipovaMA, Nazarbayev University
Malika BassarovaMA, University of Vienna

Faculty & Post-Doctoral Collaborators

Esther SongUniversity of Bergen
Assylzat KarabayevaKIMEP University
Nygmet IbadildinKIMEP University
Anar ShaikenovaKIMEP University
Jiwon LeeSeoul National University
Allan Benedict C. SolacitoNU Clark

Reidhead · AI-Human Workflow · Ewha, October 30, 202623 / 23