The 429 Is a UX Problem: Shipping an LLM Demo on Free-Tier Keys

Follow-up to Zero Fabrication by Construction.

My resume tailor runs four model calls per request on a free-tier API key, in public. Rate limits aren't an edge case there — they're the weather. The lazy move is to let the 429 bubble up as a 500 and a spinner that dies. I decided the error path deserved the same engineering as the happy path.

Three problems, in the order I hit them.

Problem 1: you can't handle the 429 you can't find

The intake stage runs two parsers concurrently under ADK's ParallelAgent, which uses an asyncio TaskGroup. So when Gemini throttles, the rate-limit error doesn't arrive as a clean exception — it arrives buried inside a BaseExceptionGroup, and chained through __cause__/__context__ on top of that. str(exc) shows you the wrapper and none of the truth.

main.py

def _exc_text(exc: BaseException | None, depth: int = 0) -> str:
    """Flatten an exception tree into one string. The rate-limit error surfaces
    inside a BaseExceptionGroup (ParallelAgent runs the parsers in a TaskGroup)
    and is also chained via __cause__/__context__, so str(exc) alone misses it."""
    if exc is None or depth > 6:
        return ""
    parts = [str(exc)]
    for sub in getattr(exc, "exceptions", None) or ():
        parts.append(_exc_text(sub, depth + 1))
    parts.append(_exc_text(getattr(exc, "__cause__", None), depth + 1))
    parts.append(_exc_text(getattr(exc, "__context__", None), depth + 1))
    return "\n".join(p for p in parts if p)

Recursive flatten, then string-match for the throttle markers. Inelegant? Sure. But the elegant version — catching a typed exception — doesn't exist when the error crosses an agent framework, an SDK, and a task group on the way up.

Problem 2: the API's retry hint lies

Gemini's free tier has two different 429s that look identical from orbit:

a per-minute throttle — the API's retryDelay is honest, retry in ~60s;
the daily quota — resets at midnight US Pacific, but the API still hands you a tiny retryDelay as if seconds from now would help.

Relaying that second retryDelay to a user is a fabrication. (Sound familiar? Same invariant as the last post, different layer: don't let the system say things that aren't true.) So the handler classifies the two cases and computes the real "ready at" time itself:

main.py

is_daily = "perday" in low.replace("_", "").replace(" ", "")

if is_daily:
    pacific = ZoneInfo("America/Los_Angeles")
    tomorrow = (now.astimezone(pacific) + timedelta(days=1)).replace(
        hour=0, minute=0, second=0, microsecond=0
    )
    ready = tomorrow.astimezone(timezone.utc)   # midnight Pacific, not "retry in 8s"
else:
    match = re.search(r"retrydelay['\"]?\s*[:=]\s*['\"]?(\d+)", low)
    ready = now + timedelta(seconds=int(match.group(1)) if match else 60)

The response is a proper 429 with a Retry-After header and a structured body: scope (short vs daily), a human message, and an ISO ready_at timestamp.

Problem 3: "come back tomorrow" is still a bad answer

A countdown is honest, but a working alternative is better. The whole agent tree is built per provider — same composition, different models — so there's a second lane on the highway:

agents.py

GEMINI_LITE, GEMINI_STRONG = "gemini-2.5-flash-lite", "gemini-2.5-flash"
GROQ_LITE, GROQ_STRONG = "groq/openai/gpt-oss-20b", "groq/openai/gpt-oss-120b"

Groq rides in through ADK's LiteLLM wrapper. The non-obvious constraint: the fallback models must support strict structured output, because every agent hand-off is a Pydantic output_schema. A fallback that returns prose isn't a fallback, it's a different product. Groq's gpt-oss models qualify; that's why they're the alternate, not whatever's cheapest.

The default provider mode is auto: try Gemini, and if — and only if — the failure classifies as a rate limit, silently retry the whole pipeline on Groq.

main.py

for index, name in enumerate(order):          # e.g. ["gemini", "groq"]
    try:
        result = await run_pipeline(resume_text, jd_text, user_id, provider=name)
        result["provider_used"] = name
        result["fell_back"] = index > 0
        return result
    except Exception as exc:
        info = _rate_limit_info(exc)
        if info is None:
            raise                              # real bugs are never masked by a fallback
        last_info = info

Two details I care about:

Only rate limits trigger fallback. A genuine error re-raises immediately. Falling back on every exception would hide bugs behind a provider switch — you'd never see them again.
The response admits it fell back. provider_used and fell_back come home with the result. No silent substitution.

And if the user pinned a specific provider and it's throttled? The 429 body names the untried alternate ("alternate": "groq", plus a human label), so the frontend can render a one-tap escape hatch:

⚡ Use Groq now

One click sets the provider and re-runs. The retry button stays disabled while a live countdown ticks toward ready_at in the user's local time — the UI physically won't let you hammer a key that isn't ready.

The shape of it

flowchart LR
    req[request] --> gem[Gemini pipeline]
    gem -->|ok| done[result + provider_used]
    gem -->|429?| cls{throttle or daily quota?}
    cls -->|"auto mode"| groq[Groq pipeline, same agents]
    groq -->|ok| done2["result + fell_back: true"]
    groq -->|429| honest["429 · ready_at + countdown + alternate"]
    gem -->|real error| boom[500 — never masked]

Classify the 429, fall back on the same agents, or answer honestly.

None of this is glamorous. It's string-matching exception soup, timezone math, and a disabled button. But it's the difference between a demo that works in the screenshot and a demo that works when a stranger clicks the link at the exact minute the quota dies.

The last post was about not lying about me. This one is about not lying about the system. Same invariant, one layer down.