A Competitive Intelligence Agent That Reads the Pricing Page

Marketing copy about competitors ages badly. An agent that re-reads their actual pages on a schedule doesn't.

Profile picture of Serply
Serply
A diff view comparing two versions of a competitor pricing page

Competitive intel decks go stale the week they’re made. The underlying problem is that they’re snapshots of pages that keep changing, assembled by hand, and nobody wants to redo the work monthly.

The automatable version has three parts: find the pages, read them, and diff against last time. Only the third part is really about competition — the first two are search and scraping.

Finding the pages

You could hard-code URLs, and for a stable competitor set you probably should. But pricing pages move, and a search step keeps the pipeline from silently reading a 404 for six months:

import os
import requests
from urllib.parse import quote_plus

API_KEY = os.environ["SERPLY_API_KEY"]


def search(query: str, num: int = 10) -> list[dict]:
    resp = requests.get(
        f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={num}",
        headers={"X-Api-Key": API_KEY, "X-Proxy-Location": "US"},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json().get("results", [])


def find_page(company: str, domain: str, kind: str = "pricing") -> str | None:
    hits = search(f"site:{domain} {kind}", num=10)
    for h in hits:
        link = h.get("link", "")
        if kind in link.lower():
            return link
    return hits[0]["link"] if hits else None

Results come back under results with title, link, description, and position. Preferring a URL that literally contains “pricing” over the top-ranked result is a small heuristic that avoids a lot of blog posts about pricing.

Reading them

def read(url: str) -> str | None:
    try:
        resp = requests.post(
            "https://api.serply.io/v1/request",
            headers={"X-Api-Key": API_KEY, "Content-Type": "application/json"},
            json={"url": url, "response_type": "markdown"},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.text
    except requests.RequestException:
        return None

Markdown is the right mode here and not just for token economy. Pricing pages are heavily styled, often with numbers injected by JavaScript into deeply nested markup; the markdown conversion flattens that into something a model can read and a diff can compare. If you need the raw markup for some reason, "response_type": "full" returns JSON with the HTML under data.

Extracting structure

Free-text diffs on a marketing page are noise — a rotated testimonial shows up as a change. Extract a stable structure first:

import json
from anthropic import Anthropic

client = Anthropic()

EXTRACT = """Extract the pricing structure from this page.

Return JSON:
{
  "plans": [
    {"name": str, "price": str, "period": str|null, "included": [str], "limits": [str]}
  ],
  "free_tier": bool,
  "enterprise_contact_only": bool,
  "notes": [str]
}

Copy prices exactly as written, including currency symbols. If a value isn't on
the page, use null — do not infer it. Ignore testimonials, FAQs, and footers."""


def extract_pricing(markdown: str) -> dict:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2000,
        system=EXTRACT,
        messages=[{"role": "user", "content": markdown[:20000]}],
    )
    return json.loads(msg.content[0].text)

“Copy prices exactly as written” prevents the most annoying class of error: a model that helpfully normalizes $49/mo to 49.00 in one run and $49 in the next, producing a diff where nothing changed.

Diffing

from pathlib import Path
import hashlib

STORE = Path("competitors")
STORE.mkdir(exist_ok=True)


def snapshot(company: str, data: dict) -> tuple[bool, dict | None]:
    """Store today's extraction; return (changed, previous)."""
    path = STORE / f"{company}.json"
    previous = json.loads(path.read_text()) if path.exists() else None
    changed = previous is not None and previous != data
    path.write_text(json.dumps(data, indent=2, sort_keys=True))
    return changed, previous

Sorting keys on write is what makes the equality check meaningful. Without it, a re-ordered JSON object reads as a change every single run.

Explaining the change

A raw diff tells you a number moved. What you want is what it means:

ANALYZE = """Two snapshots of a competitor's pricing, taken weeks apart.

Describe only what actually changed. For each change, say whether it's a price
increase, a price decrease, a repackaging, a limit change, or a new plan.
Then give one sentence on the likely commercial motivation.

If nothing material changed, say "no material change" and stop."""


def analyze(company: str, before: dict, after: dict) -> str:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1200,
        system=ANALYZE,
        messages=[{"role": "user", "content":
                   f"Company: {company}\n\nBEFORE:\n{json.dumps(before, indent=2)}"
                   f"\n\nAFTER:\n{json.dumps(after, indent=2)}"}],
    )
    return msg.content[0].text

The “if nothing material changed, say so and stop” clause is what makes this runnable weekly. A model asked to analyze two nearly-identical documents will find something to say about them unless you give it permission not to.

The run

COMPETITORS = [
    {"name": "acme", "domain": "acme.example"},
    {"name": "globex", "domain": "globex.example"},
]

for c in COMPETITORS:
    url = find_page(c["name"], c["domain"], "pricing")
    if not url:
        print(f"{c['name']}: no pricing page found")
        continue

    markdown = read(url)
    if not markdown:
        print(f"{c['name']}: could not read {url}")
        continue

    data = extract_pricing(markdown)
    changed, previous = snapshot(c["name"], data)

    if changed:
        print(f"\n=== {c['name']} changed ===\n{analyze(c['name'], previous, data)}")

A note on fairness

It’s worth being deliberate about what this pipeline is for. Reading a competitor’s public pricing page is ordinary market research — the same thing a salesperson does manually before a call. Keep it to public pages, respect the rate you’d consider reasonable if someone pointed it at you, and don’t build the version that tries to get behind a login. The intelligence value is in the tracking over time, not in access you weren’t given.