Step 4 · Extract and Score

Read the reference lists of the seeds and the strongest search results, match each reference to a bibliographic record, and score the cited works on the same scale as the search results.

Part 1 · Step 4 of Steps 0–5
Output · a scored pool of 3,684 cited works

Keyword searches find works that use the review's vocabulary. They miss the older books and articles that a field takes for granted, works written before the vocabulary settled, and works in venues the journal lists do not cover. The references of the seeds are the most direct route to them, because the seeds' authors have already judged them relevant.

Section 1Why follow citations

A work cited by several seeds is part of the conversation the review is joining, whatever words it uses. Following citations is also how a reviewer finds the classics of a field, the books on market-preserving federalism or embedded autonomy that every later article cites and few keyword searches return. The agent does in one pass what a reviewer would do by hand, reading every reference list and tallying how often each work is cited.

Section 2The source set

The agent extracted references from the seeds and reference documents and from the 50 highest-scoring search results of Step 3, 171 source records in all. A source needs a PDF, or a DOI through which its reference list can be looked up, so 130 were eligible, and 98 of these yielded references.

Table 4.1Sources of extracted references
SourcesRecords
Seeds and reference documents (comparison corpus)121
Top keyword candidates from Step 350
Source records171
with a PDF or DOI130
that yielded references98
From 08_extract-citations, run again on 12 October 2026 to replace 05_extract-citations.

Section 3Extracting and matching references

The agent read 4,005 references from the reference lists and then tried to match each one to a Crossref record, comparing titles, authors and years. A match supplies a DOI and clean metadata. A reference with no confident match is kept, with the details the citing article gives, since many books, chapters and non-English works have no Crossref record.

Table 4.2Matching the extracted references to Crossref
OutcomeReferences
Matched to a Crossref record924
No confident match2,913
Too little information to search160
Ambiguous, unresolved or failed8
Extracted references4,005
A match required a match score of at least 0.85. After the references were merged into works, 3,685 cited works were new to the review, 693 of them with a validated Crossref record, and 73 matched a seed.
Why so few matches

Fewer than a quarter of the references matched a Crossref record. Crossref covers books, working papers and Chinese and Korean articles poorly, and the seeds of this review cite many works of these kinds. A low match rate is a property of the literature, not a failure of the extraction, but it means that the cited works have much less metadata than the search results.

Section 4Scoring the cited works

Before scoring, the agent looked up missing abstracts and metadata. This raised the number of cited works with an abstract from 183 to 463. It then scored the 3,684 cited works that had a title with the same method as Step 3. With a fixed cutoff of 0.90, 26 cited works were recommended. Another 13 scored above the cutoff but failed the evidence gate, which requires an abstract or citation evidence before a score is trusted.

Table 4.3The citation pool after scoring
MeasureCount
Cited works scored3,684
with an abstract after enrichment463
above the 0.90 cutoff and passing the evidence gate26
above the cutoff but failing the evidence gate13
From 09_score-extracted-references.
A text score undervalues cited works

Most cited works have a title and nothing else, and a title says little about similarity to the seeds. Selecting the citation pool by score alone would keep only 26 works and drop the classics the seeds rely on. That is why Step 5 also selects by how often the seeds cite a work, not only by how it scores.

  • raw_references.csvEvery reference read from the source records, with the source it came from.08_extract-citations
  • citation_edges.csvWhich source cites which work, used to count citations by seeds in Step 5.08_extract-citations
  • ranked_articles.csvOne row per cited work, with its signals, combined score and evidence gate.09_score-extracted-references

Terminology

Source record
An article whose reference list is read, here a seed, a reference document or a top search result.
Cited work
A work that appears in the reference list of at least one source record.
Crossref match
A reference linked to a Crossref record, which supplies its DOI and standard metadata.
Enrichment
Looking up missing abstracts and metadata for candidates before they are scored.
Evidence gate
A check that a candidate has an abstract or citation evidence, so that a high score does not rest on a title alone.