Retry Logic That Doesn't Make Things Worse
The naive retry loop turns a brief rate limit into a sustained one. Getting backoff right takes about forty lines.
- Classify before you retry
- Exponential backoff with full jitter
- The wrapper
- Watch the budget headers instead
- Circuit breaking for sustained failure
- Putting it in the tool
- What to instrument
Everyone writes the same first retry loop: catch the exception, sleep one second, try again, three times. It works in testing and it’s actively harmful in production, because the failures that matter aren’t independent. When you’re rate limited, so is every other worker, and a fixed-delay retry synchronises them all into a thundering herd against a service that just told you to slow down.
Classify before you retry
Not every error deserves a retry. Some are permanent, and retrying them wastes budget and delays the real answer:
from dataclasses import dataclass
@dataclass
class Decision:
retry: bool
reason: str
honor_retry_after: bool = False
def classify(status: int | None, exc: Exception | None) -> Decision:
if exc is not None:
name = type(exc).__name__
if "Timeout" in name or "Connection" in name:
return Decision(True, "network")
return Decision(False, f"unexpected: {name}")
if status == 429:
return Decision(True, "rate_limited", honor_retry_after=True)
if status in (500, 502, 503, 504):
return Decision(True, "server_error")
if status == 404:
return Decision(False, "not_found")
if status == 422:
return Decision(False, "bad_request")
if status and 400 <= status < 500:
return Decision(False, f"client_error_{status}")
return Decision(False, "ok")422 is the one people retry by mistake. It means the request itself is malformed — with this API, usually an unencoded query or a ?q= querystring where the query belongs in the path. Retrying it identically will fail identically, forever.
404 is similarly terminal for a given URL. Retry it and you’ve spent three calls confirming the same thing.
Exponential backoff with full jitter
import random
import time
def backoff_delay(attempt: int, base: float = 0.5, cap: float = 30.0) -> float:
"""Full jitter: uniform in [0, min(cap, base * 2**attempt)]."""
ceiling = min(cap, base * (2 ** attempt))
return random.uniform(0, ceiling)Full jitter — a uniform draw from zero to the ceiling, not the ceiling itself — is the version that actually decorrelates concurrent clients. “Exponential backoff” with a deterministic delay keeps every worker in lockstep; they all wait 4 seconds and all retry at the same instant.
Randomising across the whole range means retries spread out, and the service sees a smooth ramp instead of repeated spikes.
The wrapper
import requests
from urllib.parse import quote_plus
API_KEY = os.environ["SERPLY_API_KEY"]
class RetryExhausted(Exception):
pass
def request_with_retry(
method: str, url: str, *, max_attempts: int = 4,
timeout: float = 30.0, **kwargs
) -> requests.Response:
last = None
for attempt in range(max_attempts):
try:
resp = requests.request(method, url, timeout=timeout, **kwargs)
decision = classify(resp.status_code, None)
if not decision.retry:
return resp
last = f"HTTP {resp.status_code} ({decision.reason})"
delay = backoff_delay(attempt)
if decision.honor_retry_after:
retry_after = resp.headers.get("Retry-After")
if retry_after:
try:
delay = max(delay, float(retry_after))
except ValueError:
pass
except requests.RequestException as e:
decision = classify(None, e)
if not decision.retry:
raise
last = f"{type(e).__name__}: {e}"
delay = backoff_delay(attempt)
if attempt == max_attempts - 1:
break
time.sleep(delay)
raise RetryExhausted(f"{max_attempts} attempts failed. Last: {last}")Note it returns the response for non-retryable statuses rather than raising. A 404 is information the caller wants, not an exception — the agent should be told the page doesn’t exist so it tries another source.
Honouring Retry-After when present, but never going below the jittered delay, means you respect an explicit instruction without collapsing back to synchronised retries when the header is absent.
Watch the budget headers instead
Retrying a 429 is recovery. Not hitting it is better, and the API tells you where you stand:
class RateTracker:
def __init__(self):
self.limit: int | None = None
self.remaining: int | None = None
def update(self, resp: requests.Response) -> None:
try:
self.limit = int(resp.headers["x-ratelimit-requests-limit"])
self.remaining = int(resp.headers["x-ratelimit-requests-remaining"])
except (KeyError, ValueError):
pass
@property
def pressure(self) -> float:
if not self.limit:
return 0.0
return 1.0 - (self.remaining or 0) / self.limit
def throttle_delay(self) -> float:
"""Slow down as the budget depletes, before hitting the wall."""
p = self.pressure
if p < 0.8:
return 0.0
if p < 0.95:
return 0.25
return 1.0Every response carries x-ratelimit-requests-limit and x-ratelimit-requests-remaining. Reading them and easing off at 80% consumption is far better than sprinting into a 429 and backing off afterwards — you never lose a request, and the slowdown is gradual instead of a cliff.
Circuit breaking for sustained failure
If the service is genuinely down, retrying every call individually turns a fast failure into a slow one across your whole fleet:
class CircuitBreaker:
def __init__(self, threshold: int = 5, cooldown: float = 60.0):
self.threshold = threshold
self.cooldown = cooldown
self.failures = 0
self.opened_at: float | None = None
def allow(self) -> bool:
if self.opened_at is None:
return True
if time.monotonic() - self.opened_at >= self.cooldown:
self.opened_at = None # half-open: let one through
self.failures = self.threshold - 1
return True
return False
def record(self, ok: bool) -> None:
if ok:
self.failures = 0
self.opened_at = None
else:
self.failures += 1
if self.failures >= self.threshold:
self.opened_at = time.monotonic()Half-open recovery matters: after the cooldown, let exactly one request through rather than reopening the floodgates. If it succeeds, the counter resets; if it fails, you’re closed again with one wasted call instead of hundreds.
Putting it in the tool
tracker = RateTracker()
breaker = CircuitBreaker()
def search(query: str, num: int = 10) -> str:
if not breaker.allow():
return ("The search service is currently unavailable. Answer from what "
"you already have, and state that you could not verify further.")
delay = tracker.throttle_delay()
if delay:
time.sleep(delay)
try:
resp = request_with_retry(
"GET",
f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={num}",
headers={"X-Api-Key": API_KEY, "X-Proxy-Location": "US"},
)
except RetryExhausted as e:
breaker.record(False)
return f"Search unavailable after retries ({e}). Proceed without it."
tracker.update(resp)
breaker.record(resp.ok)
if resp.status_code == 404:
return f'No results for "{query}". Try different keywords.'
if not resp.ok:
return f"Search failed with HTTP {resp.status_code}."
results = resp.json().get("results", [])
if not results:
return f'No results for "{query}". Try broader keywords.'
return "\n\n".join(
f"{r.get('title')}\n{r.get('link')}\n{r.get('description', '')}"
for r in results
)Every failure path returns a string the model can act on. That’s the design principle worth carrying across all of this: retries and circuit breakers are for the infrastructure, but when they’re exhausted the agent still needs to know what to do, and “proceed without it and say what you couldn’t verify” is almost always the right instruction.
What to instrument
Log the attempt count on every call, not just the final outcome. A rising average retry count is the earliest signal of a degrading dependency, and it shows up well before your error rate does — because the retries are still succeeding.