You Can Only Test What You Kept Deterministic: pytest for a LangGraph Agent

Follow-up to Cite or Refuse.

My multi-agent research assistant has a test suite that runs in about two seconds, in CI, with no API key and zero LLM calls. That's not because the tests skip the interesting parts. It's because, after the hallucination incident, the interesting parts — the guarantees — no longer live in the model.

Last post's conclusion was LLM proposes, code disposes: the grounding gate, the loop termination, the routing all became plain Python. This post is the payoff of that decision. Every invariant you move out of the prompt becomes a function you can pytest.

Code: github.com/ki11e6/langgraph-multi-agent

What must be true no matter what the model says

Before writing tests, I listed what the graph — not the model — is responsible for:

An ungrounded draft never ships. No sources, no citations, a dangling citation, or a citation to a source that fetched empty → refuse.
The writer↔reviewer loop halts. ACCEPT ends it; MAX_REVISIONS ends it even if the reviewer never accepts.
The reviewer's verdict is parsed robustly — "UNACCEPTABLE" must not read as ACCEPT.
In human-in-the-loop mode, the human's decision wins, even against the model's verdict.

None of these are properties of an LLM. They're properties of three pure functions and a graph topology. So none of them need an LLM to test.

Level 1: the gates are just functions on state

LangGraph nodes and routing functions take a state object and return a dict (or an edge name). That makes the deterministic ones trivially testable:

def test_validate_refuses_dangling_citation():
    # Draft cites [^5] but only 2 sources exist -> fabricated reference.
    out = validate_node(AgentState(draft="Per the docs [^5], do X.", sources=_sources(2)))
    assert out["validation_error"]

def test_route_after_review_max_revisions_is_hard_stop():
    # Hitting the cap ends the graph even if the verdict says REVISE.
    state = AgentState(verdict="REVISE", revision_count=MAX_REVISIONS)
    assert route_after_review(state) == END

The regression test for the original incident is one line of arrange: a citation pointing at a source that isn't in state. That exact failure — the one that once shipped a polished report full of fabricated APIs — is now a red test if it ever comes back.

There's also a test I only wrote because footnote markers invited it:

def test_validate_ignores_plain_brackets():
    # Status codes / array indices must NOT be mistaken for citations.
    draft = "Returns [404] on error and reads arr[0], per the guide [^1]."
    out = validate_node(AgentState(draft=draft, sources=_sources(1)))
    assert out["validation_error"] == ""

Choosing a citation syntax a regex can check wasn't just about validation — it made the validator's own edge cases enumerable.

Level 2: fake the LLM, test the topology

Unit-testing nodes doesn't prove the compiled graph halts. For that I run the real build_graph() with a fake chat model — a plain function that routes on the system prompt:

def _role_aware_llm(messages):
    system = messages[0].content
    if "supervisor of a research team" in system:
        ...  # route: researcher -> writer -> reviewer, based on state in the prompt
    if "meticulous editor" in system:
        return AIMessage(content="Not there yet.\nREVISE")  # never accepts
    if "skilled technical writer" in system:
        return AIMessage(content="A draft paragraph grounded in a source [^1].")

monkeypatch swaps it in for get_llm, a stub replaces web_search, and then the worst-case adversary runs against the real graph: a reviewer that never accepts.

def test_graph_terminates_without_recursion_error(monkeypatch):
    ...
    final = build_graph().invoke({"messages": [HumanMessage(content="test query")]})
    assert final["revision_count"] == MAX_REVISIONS  # exact cap, no premature exit

This is the test that would have caught the near-infinite loop from the early version of this project, where "notice the ACCEPT verdict" was delegated to the LLM supervisor. And its mirror image pins the refusal path end-to-end: a writer that emits no citations gets stopped by validate even though the fake reviewer is programmed to ACCEPT — proving the gate sits upstream of the model's opinion:

final = build_graph().invoke({"messages": [HumanMessage(content="q")]})
assert final["validation_error"]        # refused
assert final.get("verdict", "") == ""   # never reached the reviewer

Level 3: the seams where providers leak in

The ugliest agent bugs I hit weren't reasoning failures — they were parsing failures at the model boundary. Each one is now a table-stakes test:

"This draft is UNACCEPTABLE." must parse as REVISE, not ACCEPT.
A genuine ACCEPT followed by a chatty closing line is still ACCEPT.
Empty feedback defaults to REVISE — the safe direction — instead of crashing.
Anthropic and Gemini can return message content as a list of blocks where Groq returns a string; one _as_text() normalizer, four tests.

If your agent supports more than one provider, the content-blocks test alone will pay for itself.

Level 4: even interrupt/resume is testable

Human-in-the-loop sounded like the part that would need a human. It doesn't. LangGraph's interrupt() surfaces as a key in the returned state, and resuming is just another invoke:

def test_hitl_human_approval_ends_the_run(monkeypatch):
    app, config, _ = _start("t-approve")           # runs until it pauses
    final = app.invoke(Command(resume={"action": "approve"}), config)
    assert final["verdict"] == "ACCEPT"  # human's call, overriding the model's REVISE

The fake reviewer in these tests always says REVISE, so the assertion proves the design intent: a human's verdict outranks the model's — and, in this mode, outranks MAX_REVISIONS too, because a human bounds the loop in a way a model can't.

What this deliberately doesn't test

No API key means no test ever checks whether the reports are good — whether retrieval found the right pages, whether the writer's prose is accurate, whether the reviewer's taste is sound. Those are eval problems: sampled, scored, non-deterministic, run occasionally against real models. Unit tests pin the floor (the system never ships an ungrounded draft, never loops forever, never misparses a verdict); evals measure the ceiling.

The mistake is using one where you need the other — an eval suite re-checking loop termination on every commit, or a unit test asserting exact LLM output strings.

You can only unit-test what you kept deterministic. Which is the best argument I know for keeping more of your agent deterministic.