A LlamaIndex Retriever Backed by Live Web Search

Implement BaseRetriever against Serply so your query engines pull from the live web instead of a stale vector index.

Profile picture of Serply
Serply
A retriever component fetching nodes from the open web

LlamaIndex is built around a clean abstraction: a retriever takes a query and returns nodes. Everything downstream — query engines, response synthesizers, routers — only cares about that contract. Which means if you implement BaseRetriever over a search API, the entire rest of the framework works unchanged, except now it’s reading the live web.

This is a better fit than it sounds. Vector indexes are excellent for a corpus you own and terrible for anything that changed this week.

The minimal retriever

import os
import requests
from urllib.parse import quote_plus

from llama_index.core.retrievers import BaseRetriever
from llama_index.core.schema import NodeWithScore, TextNode
from llama_index.core import QueryBundle

API_KEY = os.environ["SERPLY_API_KEY"]


class SerplyRetriever(BaseRetriever):
    """Retrieves nodes from live Google results via Serply."""

    def __init__(self, num_results: int = 10, proxy_location: str = "US"):
        self.num_results = num_results
        self.proxy_location = proxy_location
        super().__init__()

    def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
        q = quote_plus(query_bundle.query_str)
        resp = requests.get(
            f"https://api.serply.io/v1/search/q={q}&num={self.num_results}",
            headers={
                "X-Api-Key": API_KEY,
                "X-Proxy-Location": self.proxy_location,
            },
            timeout=30,
        )
        resp.raise_for_status()
        results = resp.json().get("results", [])

        nodes = []
        for r in results:
            node = TextNode(
                text=f"{r.get('title', '')}\n\n{r.get('description', '')}",
                metadata={
                    "url": r.get("link", ""),
                    "title": r.get("title", ""),
                    "position": r.get("position"),
                    "display_url": (r.get("metadata") or {}).get("display_url", ""),
                },
            )
            # Rank descending: position 1 scores highest.
            score = 1.0 / (r.get("position") or 1)
            nodes.append(NodeWithScore(node=node, score=score))
        return nodes

Serply embeds the query string in the path (/v1/search/q=...) rather than using query parameters. Results come back under results, each with title, link, description, and position.

Deriving the score from position gives you a sane default ordering. It’s reciprocal rank, not a similarity score, so don’t mix these nodes into a threshold-based filter tuned for embeddings — the numbers mean different things.

Snippets aren’t enough

The retriever above returns two-line snippets. For real question answering you want page text, which means a second call per result. Fetch them concurrently or the latency is unusable:

from concurrent.futures import ThreadPoolExecutor


class SerplyFullTextRetriever(SerplyRetriever):
    """Like SerplyRetriever, but fetches full page text for the top results."""

    def __init__(self, num_results: int = 10, read_top_k: int = 4, **kw):
        self.read_top_k = read_top_k
        super().__init__(num_results=num_results, **kw)

    def _read(self, url: str) -> str | None:
        try:
            resp = requests.post(
                "https://api.serply.io/v1/request",
                headers={"X-Api-Key": API_KEY, "Content-Type": "application/json"},
                json={"url": url, "response_type": "markdown"},
                timeout=60,
            )
            resp.raise_for_status()
            return resp.text
        except requests.RequestException:
            return None

    def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
        nodes = super()._retrieve(query_bundle)
        targets = nodes[: self.read_top_k]

        with ThreadPoolExecutor(max_workers=self.read_top_k) as pool:
            texts = list(pool.map(lambda n: self._read(n.node.metadata["url"]), targets))

        for node, text in zip(targets, texts):
            if text:
                node.node.text = text[:12000]
        return nodes

The scraper’s markdown mode returns the converted text directly as the body, so .text is what you want — not .json(). Requesting "response_type": "full" returns a JSON object with raw HTML under data instead, which is the right choice if you need the markup but the wrong one if you’re feeding an LLM.

Dropping it into a query engine

Nothing special required — that’s the payoff:

from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core import get_response_synthesizer

retriever = SerplyFullTextRetriever(num_results=10, read_top_k=4)

engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=get_response_synthesizer(response_mode="compact"),
)

response = engine.query("Which SERP APIs support geo-targeted proxy locations?")
print(response)

for node in response.source_nodes:
    print(node.metadata["url"])

Routing between web and local

The genuinely useful configuration is a router that picks the web retriever for time-sensitive questions and your vector index for everything else:

from llama_index.core.tools import RetrieverTool
from llama_index.core.retrievers import RouterRetriever

web_tool = RetrieverTool.from_defaults(
    retriever=SerplyFullTextRetriever(),
    description=(
        "Live web search. Use for current events, recent releases, prices, "
        "news, or anything that may have changed recently."
    ),
)

local_tool = RetrieverTool.from_defaults(
    retriever=vector_index.as_retriever(similarity_top_k=5),
    description=(
        "Internal document corpus. Use for company policies, product docs, "
        "and anything specific to our own materials."
    ),
)

router = RouterRetriever.from_defaults(retriever_tools=[web_tool, local_tool])

Write those descriptions carefully. The router is an LLM reading exactly those strings, and “searches the web” gets picked far less reliably than a description that names the situations it’s for.

Regional results

X-Proxy-Location accepts a set of country codes — US, GB, DE, FR, JP, AU, IN, and others. If your users ask locale-specific questions, exposing this as a retriever parameter is worth doing; the same query about, say, consumer protection rules returns materially different pages from a UK exit node than a US one.