Cite or Refuse: Debugging a Fluent Hallucination in a LangGraph Research Agent

I asked my own research agent a question I already knew the answer to. It handed me a polished, confident report — citing APIs that do not exist.

The system under the hood: a LangGraph StateGraph where a Supervisor routes work between a Researcher, a Writer, and a Reviewer, all sharing a typed state object, with a bounded Writer–Reviewer refinement loop. On paper, exactly the review rigor a real report needs.

The lie

The report invented a LangfuseTracer class, a graph.run(model=..., prompt=...) signature that was never real, and version numbers that don't exist. And here is the eerie part: the Reviewer smelled the inaccuracy every single round — "verify the exact env var names" — but it had no tools to verify anything. So it did the only thing it could: it polished the lie. Three rounds later, it force-accepted.

The output looked more trustworthy the longer it ran.

sequenceDiagram
    participant R as researcher
    participant W as writer
    participant V as reviewer (no tools)
    R->>W: prose summary — source URLs already gone
    W->>V: draft citing plausible Langfuse APIs
    loop 3 review rounds
        V->>W: REVISE (style and structure notes)
        W->>V: revised draft, same fabricated APIs
    end
    V-->>V: ACCEPT — fabrications ship

The failure: by the time the Writer starts, no evidence exists in state for anyone to check.

Root cause: ungrounded state, not a bad prompt

Only the Researcher ever touched reality — and it summarized the search results into prose, dropping the source URLs. That summarize step is where the evidence died. From then on the Writer and Reviewer reasoned over a lossy paraphrase with no source text in context, so the model filled the gaps from parametric memory. Iterating a writer–reviewer loop with no new evidence didn't add accuracy; it added fluency — polish on the wrong answer.

The fix: a node the LLM can't talk its way past

Three changes. Web search now returns structured sources — title, URL, snippet, and fetched page content — and those objects travel through graph state, so provenance is preserved instead of summarized away. The Writer must cite every claim with footnote markers. And between Writer and Reviewer sits a validate node: deterministic, plain Python, no LLM.

graph TD
    S([START]) --> sup[supervisor]
    sup -->|no sources in state| res[researcher]
    res -->|"sources: {title, url, snippet, content}"| sup
    sup -->|sources present, no draft| wr[writer]
    wr -->|"draft with footnote citations"| val["validate — deterministic Python gate"]
    val -->|no sources / no citations / dangling citation| REF([END: REFUSE])
    val -->|grounded| rev[reviewer]
    rev -->|REVISE, capped at 3| wr
    rev -->|ACCEPT| OK([END])

The fixed StateGraph. The fix wasn't a better prompt — it was a node the LLM can't talk its way past.

agents/graph.py

CITATION = re.compile(r"\[\^(\d+)\]")

def validate(state: AgentState) -> dict:
    """Deterministic gate between writer and reviewer. No LLM."""
    cited = {int(n) for n in CITATION.findall(state.draft)}
    if not state.sources or not cited:
        return {"validation_error": "no grounded claims"}
    if any(n < 1 or n > len(state.sources) for n in cited):
        return {"validation_error": "dangling citation"}
    return {}  # grounded -> reviewer

The same principle fixed the loop itself. An early version delegated "notice the ACCEPT verdict" to the LLM Supervisor — which caused a near-infinite loop. Now the verdict is persisted in state and routing ends deterministically, with the revision count capped at three. An optional human-in-the-loop mode (LangGraph interrupt() plus a checkpointer) can override the cap: a human bounds the loop; the recursion limit is the backstop. If nothing can be grounded at all, the tool prints INSUFFICIENT GROUNDING and exits non-zero instead of inventing an answer.

LLM proposes, code disposes.