Going Deeper Than Page One Without Wasting Requests

Most agents read the top three results. When you genuinely need depth, the way you paginate determines whether you get coverage or duplicates.

Profile picture of Serply
Serply
Search results being collected across multiple pages and deduplicated

Most of the time, depth is the wrong instinct — the answer is in the top five and reading fifty results is a waste. But some jobs genuinely need coverage: building a link list, mapping who writes about a topic, finding every vendor in a category. For those, how you go deep determines whether you get a hundred distinct sources or the same twelve domains repeated.

Ask for more in one request first

The search endpoint takes num in the path alongside the query:

import os
import requests
from urllib.parse import quote_plus

API_KEY = os.environ["SERPLY_API_KEY"]


def search(query: str, num: int = 10, location: str = "US") -> dict:
    resp = requests.get(
        f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={num}",
        headers={"X-Api-Key": API_KEY, "X-Proxy-Location": location},
        timeout=45,
    )
    resp.raise_for_status()
    return resp.json()


data = search("open source vector database", num=100)
print(len(data.get("results", [])))

One request with a high num beats several small ones: fewer round trips, one rate-limit slot, and no cross-page duplicates. Start here before building anything cleverer.

Do check what you actually got back rather than what you asked for. The number of results a page yields depends on the query, and a request for 100 can return fewer.

Query variation beats page depth

Here’s the thing that surprises people: results 50–100 for a query are usually worse than results 1–20 for a well-chosen sibling query. Depth on one phrasing returns increasingly marginal pages; breadth across phrasings returns different neighbourhoods of the index.

import json
from anthropic import Anthropic

client = Anthropic()

EXPAND = """Generate 6 search queries that together cover this topic broadly.

Vary the angle, not just the wording: different vocabulary a practitioner vs a
buyer would use, adjacent terminology, and one query aimed at comparisons or
lists. Keyword style, not sentences.

Return JSON: {"queries": [str]}"""


def expand(topic: str) -> list[str]:
    msg = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=600,
        system=EXPAND, messages=[{"role": "user", "content": topic}],
    )
    return json.loads(msg.content[0].text)["queries"]

Six queries at num=20 gives you a wider, higher-quality pool than one query at num=120, for roughly the same spend.

Collecting and deduplicating

from urllib.parse import urlparse, urlunparse, parse_qsl, urlencode
from concurrent.futures import ThreadPoolExecutor

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
            "gclid", "fbclid", "ref", "source", "sa", "ved", "usg", "ei"}


def canonical(url: str) -> str:
    try:
        p = urlparse(url)
        params = [(k, v) for k, v in parse_qsl(p.query) if k.lower() not in TRACKING]
        host = p.netloc.lower()
        if host.startswith("www."):
            host = host[4:]
        path = p.path.rstrip("/") or "/"
        return urlunparse(("https", host, path, "", urlencode(params), ""))
    except Exception:
        return url


def collect(queries: list[str], num: int = 20) -> list[dict]:
    with ThreadPoolExecutor(max_workers=4) as pool:
        pages = list(pool.map(lambda q: search(q, num), queries))

    seen: dict[str, dict] = {}
    for query, page in zip(queries, pages):
        for r in page.get("results", []):
            key = canonical(r.get("link", ""))
            if not key:
                continue
            if key in seen:
                seen[key]["found_by"].append(query)
                seen[key]["best_rank"] = min(
                    seen[key]["best_rank"], r.get("position") or 999
                )
            else:
                seen[key] = {
                    "url": r.get("link"),
                    "canonical": key,
                    "title": r.get("title"),
                    "snippet": r.get("description"),
                    "best_rank": r.get("position") or 999,
                    "found_by": [query],
                }
    return list(seen.values())

URL canonicalisation matters more than it looks. The same article arrives with different tracking parameters from different queries, and naive set-deduplication keeps all of them. Note that Serply’s video results in particular carry Google’s sa=, ved=, and usg= parameters, which is why they’re in the strip list.

Tracking found_by gives you a free quality signal: a URL surfaced by four of six independent queries is more central to the topic than one that appeared once at rank 18.

Ranking the merged pool

def rank(items: list[dict], total_queries: int) -> list[dict]:
    for i in items:
        coverage = len(set(i["found_by"])) / total_queries
        rank_score = 1.0 / (i["best_rank"] + 5)
        i["score"] = 0.6 * coverage + 0.4 * rank_score * 10
    return sorted(items, key=lambda i: -i["score"])

Weighting cross-query coverage above single-query rank is deliberate. Rank within one query reflects that query’s phrasing; appearing across several reflects the topic.

Enforcing domain diversity

Left alone, a merged pool skews hard toward a few large sites. If the goal is coverage, cap per domain:

from collections import defaultdict


def diversify(items: list[dict], per_domain: int = 3) -> list[dict]:
    counts = defaultdict(int)
    out = []
    for i in items:
        host = urlparse(i["url"]).netloc.lower().removeprefix("www.")
        if counts[host] >= per_domain:
            continue
        counts[host] += 1
        out.append(i)
    return out

Applied after ranking, so each domain contributes its best pages rather than an arbitrary three.

Knowing when to stop

For open-ended discovery, stop when new queries stop producing new domains:

def until_saturated(topic: str, max_rounds: int = 4) -> list[dict]:
    pool, domains, rounds = [], set(), 0
    queries = expand(topic)

    while rounds < max_rounds and queries:
        batch = collect(queries[:3], num=20)
        queries = queries[3:]
        new_domains = {
            urlparse(i["url"]).netloc.lower().removeprefix("www.") for i in batch
        } - domains

        pool.extend(batch)
        domains |= new_domains
        rounds += 1

        if len(new_domains) < 3:
            break

    merged: dict[str, dict] = {}
    for i in pool:
        merged.setdefault(i["canonical"], i)
    return diversify(rank(list(merged.values()), rounds * 3))

The saturation check is what keeps this from being an expensive loop. When a round of queries adds fewer than three domains you’ve found what this topic has, and further requests are buying you the long tail of scraper sites.