Classifying competitor page diffs with an LLM

RivalSweep watches competitors’ public pages, pricing, changelog, integrations and positioning, and turns the changes that matter into alerts and a weekly briefing. A language model is part of that, but its job is deliberately small: given the diff of one page, say what category of change it is, how important it is, summarise what changed, and offer one sentence of interpretation. Everything before and after that call is deterministic code, and the model is the last step in the pipeline, not the first.

I build and run RivalSweep alone. Most of the code was written with AI coding assistants, with me owning the design and operating it in production. This note is about the model inside the product, and about the five weeks of production that shaped how it is fenced in.

RivalSweep is early. It is still onboarding its first customers, and the filtering and classification described here are revised constantly, driven by my own use of the product on a public watchlist and by feedback from the first users. Read this as a snapshot of a pipeline that changes every week, not as a finished design. The dated incidents further down are what that revision looks like in practice.

The model is the last step

Step	What happens	Deterministic?
Schedule	Every five minutes, pick approved pages that are due	Yes
Fetch	Per-host throttle, SSRF check, `robots.txt` (fails closed), then HTTP or a headless browser	Yes
Normalise	HTML to text plus heading sections; scripts, navigation, footers dropped	Yes
Hash and filter	Same hash, truncated-tail drift, whitespace-only change, revert to a known version: stop here	Yes
Snapshot and diff	Store the capture; line diff plus a structured section diff with observed price pairs	Yes
Classify	One model call, or the heuristic classifier	The model
Route	High importance or low confidence goes to a review queue; alert queued on commit	Yes
Brief	Weekly briefing from the stored, classified events	Mostly

Classification runs inside the same database transaction as the crawl, exactly once per pair of snapshots. An event is created with get_or_create on the page and the snapshot, and the model is called only if the event is new. A retried crawl can never classify, and bill, the same change twice.

What never reaches the model

Most of the work of keeping the classifier honest happens before it is called. These rules exist because each one failed in production first:

Same hash as the previous snapshot. Nothing to do.
Truncated-tail drift. Large pages are captured up to a byte cap. When the head of a page grows by a few bytes, the cut moves, entries fall off the tail, and the diff reads as a deletion the site never made. The first change event this product ever produced was exactly that, a false deletion on a changelog page (2026-09-05). Content beyond the cut is not observed fact, so a change that lives only there is not evidence.
Whitespace-only differences. When the normaliser changes, every content hash changes. The pipeline re-baselines silently instead of alerting every customer about every page after a deploy.
Reverts and A/B variants. A capture whose content matches any of the page’s last ten snapshots is stored as evidence but filed as “Reverted to the version seen on …”, low importance, no model call, no alert. One pricing page alternated between two layouts for days and each flip was classified as a new pricing overhaul: about 25 high-importance events for one real change (2026-10-11).
Budget exhausted. Each workspace has a monthly AI budget. When it is spent, the heuristic classifier takes over and the row says so.

What the model sees

The model does not get the page, and it does not get a unified diff with context. It gets:

the lines removed and the lines added, each trimmed to half of a character budget;
the page type (pricing, product, integration, positioning, content) and the URL;
up to six observed price pairs, extracted by the structured diff and labelled as factual. Pairs are matched by plan label, not by position in the page. Positional matching once read a relocated banner as “$100 → $0” (2026-10-12);
all crawled text wrapped in explicit untrusted-content markers, with the instruction not to follow anything inside them.

The system prompt is stable, with no dates or per-request values, and brand-neutral:

You classify changes detected on public competitor web pages for a
competitive-intelligence product. You receive the textual diff of one
monitored page (lines removed, lines added) and must produce one JSON
classification.

Field rules:
- category: pricing | product | integration | positioning | content | noise.
- importance: "high" only for changes to customer-facing strategy (price
  moves, plan restructuring, major launches); "medium" for notable product,
  integration or positioning updates; "low" for routine content edits; pick
  "noise" as category for cosmetic or boilerplate churn.
- title: short and factual, at most 90 characters.
- summary: describe only what observably changed in the diff. If observed
  price pairs are provided, restate them exactly. Never speculate here.
- possible_impact: one sentence of competitive interpretation. This field is
  the only place for inference and is labeled as such in the product.
- confidence: your confidence in the category, between 0 and 1.
- evidence.before / evidence.after: short verbatim quotes from the removed /
  added lines that best support the classification (empty string if none).

And the output contract:

{
  "category": "pricing",
  "importance": "high",
  "title": "Pro plan price increased",
  "summary": "The monthly Pro plan changed from EUR 49 to EUR 69.",
  "possible_impact": "The competitor may be moving upmarket.",
  "confidence": 0.94,
  "evidence": { "before": "Pro — EUR 49 / month", "after": "Pro — EUR 69 / month" }
}

One field for facts, one field for opinions

The split between summary and possible_impact is the most important design decision in the feature. The summary may only describe what is in the diff. The impact field is the only place the model is allowed to infer, and the product labels it “interpretation, not fact” everywhere it appears: alert emails, Slack, the change page, the public showcase.

Two consequences of taking that seriously:

The model’s evidence quotes are never shown to customers. They are parsed and validated, but what a customer sees as “before” and “after” comes from the deterministic diff: the first changed lines and the extracted price pairs. A quote the model may have polished is not evidence, even when it reads better.
Evidence comes first in every alert. Price pairs and changed sections at the top, the AI summary and impact after, each labelled. A price pair with no “before” is rendered as “now $60”, never as an arrow from an empty cell, because that would be a claim the evidence cannot support.

Confidence is self-reported by the model, range-checked, and not calibrated. It is used for one thing: routing anything below 0.75 to the review queue.

The classifier sits behind a small provider abstraction selected by an environment variable. The Anthropic path uses server-enforced JSON schema output. The OpenAI-compatible path, which is how DeepSeek-class models are reached through a gateway, uses JSON-object mode, spells the required keys out in the prompt, and validates the shape on our side. Both check the stop reason: a truncated or refused answer is an invalid classification, not a partial one.

Every provider shares one contract: classify() never raises. Bad JSON, an unknown category, an out-of-range confidence, an API error, a timeout: all of them fall back to the heuristic classifier, a regex and keyword classifier with page-type priors. The crawl is never blocked on the model, and the fallback is visible afterwards because the ledger row says heuristic.

The heuristic is deliberately conservative. On a pricing page it only says “high” when a price was observed moving: an amount on both sides of a pair, and different. “Any digit in the diff” used to count, and it filed a relocated banner and a still-compressed response body as high-priority alerts on the same day (2026-10-12).

One detail that cost a morning: reasoning models bill their thinking as output tokens. One pricing diff spent 367 of its 528 completion tokens reasoning before writing any JSON, so the output cap for that path is four times what the JSON needs.

Between the model and the inbox

Review routing. High importance, or confidence below the threshold, goes to a review queue. Everything else is published automatically.
Alerts fire at classification time, labelled as pending review. For the first five weeks they fired only after an operator approved the event, in a review queue no customer knew existed: 43 high-importance events, 5 alerts (fixed 2026-10-11).
Feedback. If the last three reviewed events on a page were all dismissed, the next classification on that page is downgraded to noise before routing. Pricing is exempt: repetition is not proof that a pricing page has stopped mattering.
Output sanitiser at the notification boundary. Model-derived text is stripped of control characters, every URL that is not a deep link into the app is replaced with “[link removed]”, and lengths are capped. A competitor page cannot smuggle a link into a customer’s inbox through the model.

What it costs

Token counts come from the API’s usage block, never from a local tokeniser. Every metered call writes one ledger row: provider, model, prompt version, tokens, cost, and the parsed output. Rows hang off exactly one of a change event, a weekly report or a workspace, enforced by a check constraint, and all of them count against the same per-workspace monthly budget.

Two pricing lessons that are not obvious until money moves:

Price by seller, not by model id. The same model id costs different amounts bought directly or through a gateway. The price table is keyed by who sold the call, and the provider’s own reported cost wins over our table when it is present. Before that fix, calls bought through the gateway would have been metered at 2.5× their real cost, straight out of a customer’s budget. The gateway’s reported cost masked it; the day a provider stops reporting one, the table is what bills the customer.
Exact match before prefix match. deepseek-v4-flash is a prefix of deepseek-v4-flash-free. Prefix-first lookup would have invoiced a customer for free inference.

Measured on the live API: one classification on deepseek-v4-flash through the gateway costs about $0.00008, and one Sonnet-class impact pass about $0.0035. Two months of production AI, every call included, cost $0.07. The budget machinery exists for margin at scale, not for today’s bill.

Evaluation, honestly

There is no golden dataset yet. What exists:

Schema contract tests and fake clients that exercise every fallback path offline: bad JSON, wrong category, API error, bad stop reason, missing key.
Regression tests from real incidents for the deterministic layers: the truncation rule, revert suppression, price pairing, the glued-words bug in the normaliser, feedback bias, alert dispatch.
Offline by construction. The test runner forces the offline providers. It had to: the day a developer environment was pointed at a real gateway, the crawl-pipeline suites started making billed API calls and the run went from 58 seconds to hanging past two minutes.
One model-level evaluation. When classification moved from Claude Haiku to deepseek-v4-flash, the 16 stored production diffs were replayed against the new model and compared with Haiku’s labels: zero heuristic fallbacks, 13 of 16 same category, five of Haiku’s six pricing/high preserved, 9.2× cheaper in practice (less than list price suggests, because of the reasoning tokens). The rule I applied: a cheaper classifier that misses one competitor’s price change is a loss, not a saving.

Every ledger row stores the prompt version, so a replay like that is always possible. What is missing, in order of value: a labelled set of real diffs, precision and recall on pricing/high specifically, confidence calibration, and quality over a longer run than 16 diffs.

What went wrong

2026-09-05. The first event ever was false: a truncated changelog read as a deletion.
2026-09-09. The Anthropic SDK appends /v1/messages to the base URL and the OpenAI SDK appends /chat/completions, so one gateway URL worked for one and produced a 404 for the other. Every caller of the failing roles was best-effort and swallowed the error, so the output would have quietly degraded with nothing in the logs. It was caught by running the real code against the gateway before production was switched over, which is the only reason it is a paragraph and not an incident. Same day: tests were spending real money.
2026-10-11. The A/B flip-flop, and the alert gate nobody knew about.
2026-10-12. Positional price pairing, a compressed body that was never decoded, inline elements joined into “MonthlyYearly” and quoted as evidence, and a heuristic title so generic that three distinct changelog entries read as one event repeated three times.

None of these were model failures. The model has been the most reliable component in the pipeline. The failures were in what was fed to it and in what was done with its answer.

What I would tell someone building this

Put the model last. Make everything before it deterministic, hashable and testable, and let the model see only what survives.
Give the model one job and one field for opinions, and label that field.
Never show the model’s quotes as evidence. Show the diff.
Log enough to replay: prompt version, model, tokens and cost on every row. Your first evaluation set will be your production history.
Make the fallback boring and visible. A regex classifier that says “heuristic” in the ledger beats a retry loop.
Best-effort paths need a signal. An error that is swallowed to protect the crawl can hide a broken feature for weeks.

The same rules, applied to a support assistant that reads customer data and could say something wrong to a person, are in the note on TriBathon’s support assistant.