You Can't Debug an Agent You Can't See
When a research agent returns a bad answer, the bug is almost never in the model. It's in what it searched for and what came back.
- Record the call, not just the outcome
- Wrapping the tools
- The scrape side
- Derived signals worth alerting on
- Sample the full trace, keep every summary
- What this actually buys you
A user reports that the agent gave a wrong answer. You have the question and the answer. You do not have the four queries it ran, the twelve results it saw, the two pages it read, or the one page whose fetch timed out silently.
Without those, every debugging session is a re-run and a guess. With them, most bugs are obvious in thirty seconds — and the pattern is consistent: the model reasoned fine over bad retrieval.
Record the call, not just the outcome
import time
import uuid
import json
from dataclasses import dataclass, field, asdict
from typing import Any
@dataclass
class ToolCall:
call_id: str
trace_id: str
tool: str
args: dict
started_at: float
duration_ms: float | None = None
status: str = "pending" # ok | error | empty | rate_limited
http_status: int | None = None
result_size: int | None = None
result_count: int | None = None
urls_returned: list[str] = field(default_factory=list)
error: str | None = None
class Trace:
def __init__(self, question: str, user_id: str | None = None):
self.trace_id = str(uuid.uuid4())
self.question = question
self.user_id = user_id
self.started = time.monotonic()
self.calls: list[ToolCall] = []
self.answer: str | None = None
def start(self, tool: str, args: dict) -> ToolCall:
call = ToolCall(
call_id=str(uuid.uuid4())[:8], trace_id=self.trace_id,
tool=tool, args=args, started_at=time.monotonic(),
)
self.calls.append(call)
return call
def to_json(self) -> str:
return json.dumps({
"trace_id": self.trace_id,
"question": self.question,
"user_id": self.user_id,
"total_ms": round((time.monotonic() - self.started) * 1000),
"calls": [asdict(c) for c in self.calls],
"answer": self.answer,
})urls_returned is the field people leave out and then wish they had. When an answer cites a source that looks wrong, the first question is always “did that URL actually come back from a search, or did the model make it up?” — and this answers it without a re-run.
Wrapping the tools
import requests
from urllib.parse import quote_plus
API_KEY = os.environ["SERPLY_API_KEY"]
def traced_search(query: str, trace: Trace, num: int = 10) -> str:
call = trace.start("web_search", {"query": query, "num": num})
try:
resp = requests.get(
f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={num}",
headers={"X-Api-Key": API_KEY, "X-Proxy-Location": "US"},
timeout=30,
)
call.http_status = resp.status_code
if resp.status_code == 429:
call.status = "rate_limited"
call.duration_ms = round((time.monotonic() - call.started_at) * 1000)
return "Rate limited. Wait a few seconds before searching again."
resp.raise_for_status()
results = resp.json().get("results", [])
call.result_count = len(results)
call.urls_returned = [r.get("link", "") for r in results]
call.status = "ok" if results else "empty"
text = "\n\n".join(
f"{r.get('title')}\n{r.get('link')}\n{r.get('description', '')}"
for r in results
) or f'No results for "{query}".'
call.result_size = len(text)
return text
except Exception as e:
call.status = "error"
call.error = f"{type(e).__name__}: {e}"
return f"Search failed: {e}"
finally:
call.duration_ms = round((time.monotonic() - call.started_at) * 1000)Distinguishing empty from ok is the single highest-value distinction in this whole schema. A zero-result search is not an error — nothing throws, the agent gets a polite message, and the answer degrades quietly. It’s the most common invisible failure in production research agents, and you cannot see it in an error rate.
Same for rate_limited. A 429 handled gracefully is still a 429, and a spike in them explains a sudden drop in answer quality that would otherwise look like model regression.
The scrape side
def traced_read(url: str, trace: Trace, max_chars: int = 15_000) -> str:
call = trace.start("read_page", {"url": url})
try:
resp = requests.post(
"https://api.serply.io/v1/request",
headers={"X-Api-Key": API_KEY, "Content-Type": "application/json"},
json={"url": url, "response_type": "markdown"},
timeout=60,
)
call.http_status = resp.status_code
resp.raise_for_status()
text = resp.text
call.result_size = len(text)
call.status = "ok" if len(text) > 200 else "empty"
if len(text) > max_chars:
return text[:max_chars] + f"\n\n[Truncated at {max_chars}.]"
return text
except Exception as e:
call.status = "error"
call.error = f"{type(e).__name__}: {e}"
return f"Could not fetch {url}. Try another source."
finally:
call.duration_ms = round((time.monotonic() - call.started_at) * 1000)Markdown mode returns text in the body, so .text — and result_size on a scrape is a genuine quality signal. A 200-character “page” is a cookie wall or a JS shell, and marking it empty rather than ok surfaces a class of silent failure that otherwise looks like the model ignoring good evidence.
Derived signals worth alerting on
Raw traces are for debugging. These aggregates are for noticing:
def summarize(trace: Trace) -> dict:
calls = trace.calls
searches = [c for c in calls if c.tool == "web_search"]
reads = [c for c in calls if c.tool == "read_page"]
queries = [c.args.get("query", "") for c in searches]
unique_urls = {u for c in searches for u in c.urls_returned if u}
cited = set(re.findall(r"https?://[^\s\)\]]+", trace.answer or ""))
return {
"trace_id": trace.trace_id,
"searches": len(searches),
"reads": len(reads),
"repeat_queries": len(queries) - len(set(q.lower() for q in queries)),
"empty_searches": sum(1 for c in searches if c.status == "empty"),
"failed_reads": sum(1 for c in reads if c.status != "ok"),
"rate_limited": sum(1 for c in calls if c.status == "rate_limited"),
"distinct_urls_seen": len(unique_urls),
"citations": len(cited),
"uncited_from_retrieval": len(cited - unique_urls),
"slowest_ms": max((c.duration_ms or 0) for c in calls) if calls else 0,
}Two of these deserve alerts.
uncited_from_retrieval counts URLs the answer cites that never appeared in any search result. Non-zero means fabricated citations, which is the failure users notice and lose trust over. It’s a set difference and it costs nothing.
repeat_queries counts the agent asking the same thing twice. A rising average means your prompt or tool descriptions are pushing the model into a rephrase loop, and it’s a leading indicator of a cost spike.
Sample the full trace, keep every summary
Full traces are large — result text, page content, all of it. Keep the summary for every request and the full trace for a sample plus every failure:
def persist(trace: Trace, summary: dict) -> None:
metrics.emit(summary)
should_keep_full = (
summary["uncited_from_retrieval"] > 0
or summary["empty_searches"] > 0
or summary["failed_reads"] > 1
or summary["searches"] > 10
or hash(trace.trace_id) % 100 < 2 # 2% sample
)
if should_keep_full:
blob_store.put(f"traces/{trace.trace_id}.json", trace.to_json())Sampling on a hash of the trace ID rather than a random draw keeps the sample stable if you re-run the persist step, and biasing the retained set toward failures means the traces you have are the ones you’ll want.
What this actually buys you
Once traces exist, the common bug reports resolve fast. “The answer was outdated” is usually an empty news search. “It cited a page that doesn’t say that” is usually a truncated read. “It’s slow” is usually one 40-second scrape in a sequential loop that should have been concurrent.
None of those are model problems, and none of them are visible without the trace.