Step 3 · Search and Score

Run every term against every journal and both open indexes, pool what the four searches find into one list of candidates, remove the seeds, and score each candidate by its similarity to the seeds.

Part 1 · Step 3 of Steps 0–5
Output · a scored pool of 19,186 keyword candidates

This is the step where an agent saves the most time. Three thousand queries, each returning up to a hundred records, are beyond anyone's patience but are routine work for an agent. The price is volume. The searches return tens of thousands of records, most of them irrelevant, and the step is only useful if it ends with a ranking the researcher can trust.

Section 1The searches

Each of the four searches queried every journal on its list with each of its ten terms, and queried Crossref and OpenAlex with the same ten terms across all journals. Journal queries kept the top 25 results for each term and journal, and open-index queries kept the top 100 for each term. The caps keep the volume manageable while letting each journal contribute its most relevant work.

Table 3.1Queries run by each search
SearchJournals × 10 termsOpen-index queriesQueries
General42020440
North Korea77020790
China1,170201,190
Vietnam62020640
Total2,980803,060
From 03_search-journals. No query failed. The open-index queries are Crossref and OpenAlex, once for each of the ten terms.

Section 2Pooling the results

The same work is often found many times, by several terms, in several searches, in a journal query and again in an open index. The agent keeps every one of these discovery events, because the number of independent routes that lead to a work is itself evidence of relevance, and then merges them into one candidate per work. Works that match a seed are set aside, since the review already has them.

Table 3.2From search results to novel candidates
StageCount
Discovery events (a work found by one query)69,001
Source records20,063
Candidates after merging duplicates19,319
Candidates that match a seed78
Novel candidates19,241
Share of novel candidates with an abstract36%
The pooled results of the four searches, checked against 251 distinct seed identities.

Section 3The special platforms

The capture guides written in Step 2 asked for 40 manual searches, the ten terms in each of CNKI, Airiti, KCI and RISS, saving the top 25 results of each to 1 Captures with a tag naming the database and term. In the worked example no captures had been imported when the searches were scored, so the special platforms did not contribute a separately scored pool. A review that relies on Chinese or Korean literature should finish the captures before scoring.

Section 4Scoring against the seeds

Every novel candidate was then scored against the comparison corpus from Step 1. The scoring was run again on 12 October 2026 with the revised comparison corpus, and that rescore replaced the first. It covered 19,186 candidates, 12,377 of them with an abstract. The score combines several signals, each turned into a percentile so that signals on different scales can be added.

Table 3.3Signals in the keyword score
SignalMeasuresPlanned weightWeight used
Centroid similarityTF-IDF similarity of the title and abstract to the weighted centre of the comparison corpus0.350.44
Nearest-seed similaritysimilarity to the closest individual seeds0.250.31
Term coveragehow many of the approved terms the title and abstract contain0.200.25
Citation overlapreferences shared with the seeds0.20not available
From 07_score-keyword-results, which superseded 04_score-keyword-results. No citation links between the candidates and the seeds were available, so citation overlap could not be computed, and the other three weights were scaled up to sum to one.
Why several signals

Similarity to the centre of the seeds rewards works about the review's core concerns and penalises works about one corner of it. Similarity to the nearest seed rewards those corners. Term coverage keeps the score tied to the vocabulary the researcher approved. Each can be fooled on its own, a long abstract can match many terms and a survey article can sit close to the centre of almost anything, but a work that scores well on all three is very likely relevant.

Section 5The cutoff

The configuration set the cutoff on the combined score at 0.90. On 12 October 2026 the researcher lowered it to 0.85 for the keyword pool. Candidates below the cutoff are not discarded. They stay in the pool, and Step 5 can still select them by another route.

Scores rank, they do not decide

A similarity score cannot tell a critique from an endorsement, or a study of North Korea from a study that mentions it once. Reading the works on either side of the cutoff is the only way to know whether the cutoff is in the right place. A short session of reading at this point saves the review from a corpus built on an arbitrary number.

  • search resultsEvery query, its results and the pooled candidate list with discovery events.03_search-journals
  • keyword scoresOne row per candidate, with each signal, the combined score and whether it passes the cutoff.07_score-keyword-results

Terminology

Discovery event
One instance of a work being returned by one query. A work found by many queries has many discovery events.
Novel candidate
A work found by the searches that is not already a seed.
TF-IDF
A way of representing a text by its words, weighting each word by how often it appears in the text and how rare it is across all texts.
Centroid
The weighted average of the comparison corpus, a single point that stands for the seeds and reference documents as a whole.
Percentile score
A signal expressed as the share of candidates it beats, so that signals measured on different scales can be combined.
Cutoff
The combined score above which a candidate is recommended, set by the researcher.