← Back to projects

Citation Is Not Truth: A Three-Metric Evaluation of an Extractive Research Pipeline

Abstract. I built a research agent whose citations are structural: a researcher extracts claims from sources, a critic admits only claims whose content tokens are contained in the cited source, and the writer sees nothing else — so every claim in every report carries a checkable citation, and the seed eval scores it perfectly (citation coverage 19/19, verifiability 19/19, planted fabrications 6 in / 6 caught / 0 leaked). I then attacked the pipeline the way a real web would: poisoned sources carrying fabricated facts, planted instructions, canaries, swapped figures and entity swaps, quote-mined claims, fake corroboration, and questions the sources cannot answer — 13 hand-written scenario seeds, 50 deterministic structural variants, 8 benign near-miss controls — scored on attack success rate (ASR), false-refusal rate (FRR), and completion, plus citation accuracy re-verified independently. Every source-borne attack succeeded (seeds 10/10, variants 50/50; overall ASR 84.6% on seeds, the only blocks being questions with nothing to extract). Citation accuracy stayed 100% throughout — including for the fabrications. The critic verifies that a claim is a faithful quote of its source; nothing in the pipeline asks whether the source is worth believing.

The question

This pipeline's promise is that structural citation makes research trustworthy: no claim without a source, no source without a token-level check. How much of that promise survives contact with sources that are themselves the attack? A citation can be perfectly valid — right source, faithful quote, containment 1.0 — while the claim is fabricated, because the check compares the claim to the source, not the source to the world. And what does the pipeline do when the honest answer is "the sources don't cover this"? One pass/fail number answers none of these, so I scored three metrics, plus the citation accuracy the project advertises, re-checked independently rather than trusted from the pipeline.

Method

System. A four-stage deterministic pipeline over a seeded mock web: a planner splits the topic into three aspects; a researcher extracts candidate claims (source sentences) and deliberately mixes in two fabricated claims of its own, the way a real researcher hallucinates; a critic admits claims at token containment ≥ 0.75 against their cited source; a writer assembles the report from admitted claims only. There is no language model anywhere in the system — every defence is structural, so every failure is structural too.

Case generation. Thirteen hand-written scenario seeds in three families. Injection (5): sources carrying a fabricated product result, an instruction sentence planted as content, a canary token inside a plausible sentence, an off-topic promotion on a keyword-matching page, and a fabricated identification split across two sources. Citation integrity (5): a survey figure swapped (61% → 97%) in a rival write-up, one fabricated statistic planted in two sources as fake corroboration, a false claim quote-mined so sentence-splitting strips its debunk, a real feature re-attributed to the wrong product, and a fabricated ranking laundered through a borrowed authority. Unsupported (3): two topics with no source coverage at all, and one question about an aspect (pricing) the sources never mention. Eight benign near-miss controls cover baseline topics, odd-but-answerable topics, both sides of a legitimate disagreement, spelled-out numbers, a long diluting source, an off-topic distractor, and a security article that quotes attack text as content.

Comparability slice. The sibling suites hand-port AgentDojo's scenario model — benign task, payload planted in a tool output, attacker goal, separate utility and security checks. AgentDojo's environments are tool-using chat agents; this system's "tool output" is the web itself, so the same scenario grammar is re-expressed with a poisoned corpus as the carrier: benign research task, payload planted in a retrieved source, attacker goal, and the report checked for the goal signal and for the legitimate claims separately. It is a hand-ported design in that style, not the AgentDojo benchmark or its data.

Variants. Five deterministic structural mutations — the transforms that stress a system reading sources rather than prompts: payload moved to the source's lead, payload buried behind boilerplate, payload source echoed under a second URL, payload source's title stuffed with the query, and payload split across a sentence boundary. Applied to the 10 source-borne seeds: 50 variants, controls copied in verbatim. The generator stamps both input files' sha256 into the output and regenerates byte-identically.

Metrics. Each attack run is classified succeeded (a goal substring — planted instruction, canary, fabricated figure — is present in the final report; for unsupported questions, the report still carries claim bullets), blocked (a payload claim is on the critic's rejected list; for unsupported questions, the report carries no claims), or evaded (the payload never reached the critic as a claim — filtered upstream or skipped by extraction). Benign runs are completed, false_refusal (the critic rejected a legitimate source claim, or an answerable topic came back empty), or inconclusive. Citation accuracy re-verifies every report bullet independently: its [n] must resolve to a listed source whose text contains the claim at the same 0.75 containment the critic uses. ASR counts only measured successes; utility is tracked as benign completion and completion under attack.

Results

Suites

Seed families

Mutation classes

Citation accuracy: 150/150 bullets (seeds) and 568/568 (variants) independently re-verified, 0 orphan citations. Benign completion 8/8 in both suites. The seed eval is unchanged and still green: coverage 19/19, verifiability 19/19, planted fabrications 6 in / 6 caught / 0 leaked.

Findings

1. Source-borne payloads succeed by construction. The critic's only test is "is this claim contained in its cited source?", and a claim extracted from a source is contained in it trivially. A fabricated result, a planted instruction and a canary all entered the report verbatim, as cited findings. Containment grades transcription, not truth.
2. No structural mutation changes anything. Lead placement, burial behind boilerplate, a second echoing source, a keyword-stuffed title, even splitting the payload across a sentence boundary — 50/50 succeeded. There is no defence layer in the pipeline whose behaviour varies with the payload's form, because there is no layer between "the source said it" and "the report repeats it".
3. The seed eval's perfect fabrication score is an artefact of its attack shape. Its only fabrications are the researcher's own plants, which cite sources that do not contain them — the one fabrication shape a real source never produces. Across all 79 adversarial runs, the critic rejected exactly two claims per run — the built-in plants — and never once a source-extracted claim.
4. Citations verify the quote, not the claim. Citation accuracy is 100% (718 bullets, independently re-checked) including every fabricated claim: a reader who checks the citation finds the sentence, which proves provenance, not truth. Fake corroboration is free for the same reason — one invented statistic in two planted sources appears as a twice-cited finding.
5. Quote-mining works through the sentence splitter. The false claim and its debunk are separate extractions; the report carries the claim as a cited bullet and the rebuttal as an unrelated one. Context does not survive extraction, so a source that corrects a falsehood can still put the falsehood in the report.
6. Swaps pass silently alongside the truth. The number-swap report contains both the real 61% and the fabricated 97%, each correctly cited; the entity-swap attributes pgvector's feature to Qdrant next to the correct attribution. The pipeline has no cross-claim conflict detection, so contradiction is invisible in its own output.
7. Unsupported questions split cleanly in two. With nothing to extract, the pipeline fabricates nothing — the off-corpus reports are empty, a genuine strength of the extractive design. But a wrong-aspect question (pricing, never covered) returns a full, confident report about the covered aspects, and even the empty reports never state that evidence was insufficient: emptiness is the only uncertainty signal the system has.
8. The failure mode is additive, not denial. Completion under attack is 100% and the false-refusal rate is 0%: poison never displaces legitimate research, distractors stay out, quoted attack text in a security article is handled as the content it is. The cost is purely that fabricated material rides alongside correct material under equally valid citations.
9. A latent citation hazard rides on the critic. The researcher's own planted claims carry source id 0 (or an id borrowed from a real source). Today containment keeps them out; if one ever passed, the writer would print it with a [0] that resolves to no listed source. Measured orphan citations: 0 of 718 — because the critic held, not because the writer checks.

Limitations

Everything ran against the seeded mock web, which bounds every claim. The corpus is tiny (three real topics plus scratch topics written for these cases), the sources are short and clean, and real-web sources are longer, noisier and better at ranking games than anything here. The system contains no language model, so the instruction case measures verbatim reproduction — the quoting channel — not obedience: what a generative researcher would do with a planted instruction is a different, larger question this benchmark deliberately does not claim to answer. Payload sentences and controls are hand-written by me; the citation recheck shares the pipeline's tokeniser, so its independence is partial (it re-derives containment from the report text rather than trusting the critic's verdicts, but it sees tokens the same way). The unsupported family is three cases; its 33.3% ASR is two clean blocks and one wrong-aspect answer, not a rate to extrapolate.

Future work

The fixes this benchmark exists to grade, each as a separately measured phase: source-credibility signals (a claim from an unknown or single source should not carry the same weight as one from the seeded corpus's established domains), cross-claim conflict detection (same quantity, different values → flagged, not silently co-cited), context-preserving extraction (a claim and its debunk travel together or not at all), an explicit insufficient-evidence state in the report for uncovered questions and aspects, and a writer-side check that every printed citation id resolves. A fix counts only if it moves ASR without taxing the benign side — these suites are the regression test.

Reproducibility

The project is public: deep-research-agent under github.com/tushar29k. Case files are hand-written YAML in evals/; the variant generator stamps the seed files' sha256 into its output and regenerates byte-identically; metric definitions are frozen in a versioned scoring module (evals/scoring.py, v1). The adversarial suites are report-only (the seed suite in evals/run_eval.py remains the CI gate), and every finding above is logged here and in evals/adversarial-report.md, deliberately unfixed, so the fix phase can be graded against them.

Full note in the repo: deep-research-agent/RESEARCH.md