How to migrate off a web scraping vendor

Most vendor migrations are treated as an integration task, scheduled accordingly, and then produce a subtle mess three months later. The integration genuinely is a day of work. The part that matters is knowing precisely how the output differs before you depend on it.

TLDR

• Run both in parallel on the same URLs and diff the outputs. This is the whole job.
• What changes is rarely the code, it is the shape of the output, and downstream systems are tuned to the old shape.
• Compare failure behaviour, not just successful pages. That is where vendors differ most.
• Migrate a slice first. A full cutover destroys your ability to attribute any change.
• Keep the old outputs, so a regression discovered later is diagnosable.

Why swapping is not just changing a URL

Both vendors return “clean markdown from a page”, so the integration is small. The differences appear underneath:

Boilerplate handling differs. One keeps the byline and the related-articles list, the other drops them. Neither is wrong. But if your chunker was tuned around the old shape, or your dedup keyed on something that has now vanished, behaviour shifts for reasons nobody will connect to the migration.

Structure preservation differs. Whether a table survives as a table, whether code blocks keep their fences, whether headings map to the same levels. These matter enormously downstream and are invisible in any summary metric.

Failure behaviour differs. This is the big one. What comes back for a blocked page, whether that is distinguishable from an empty page, and whether you are billed for it. A pipeline written against one vendor’s failure semantics will silently misclassify another’s.

Freshness and caching differ. If one serves a cached capture where the other refetches, you can see changes that are about cache windows rather than the web.

Run both, then diff

The only reliable method, and it is not expensive:

1. Take a representative slice of your real URLs. A few hundred is plenty, chosen to match your actual mix rather than your nicest pages.
2. Fetch every one through both vendors, same day.
3. Store both outputs.
4. Diff them, and read the biggest differences by hand.

You are not looking for identical output, which will not happen. You are looking to understand the differences well enough that none of them surprise you later.

Sort by difference size and read the top twenty. Almost everything you need to know is in those.

What to compare

Content overlap. Word-level F1 between the two outputs shows how much they agree. Where they disagree sharply, one of them is usually wrong and it is worth knowing which, so check those pages by hand. There is a method for this in how to measure extraction quality yourself.

Length distribution. Systematically shorter output from the new vendor means it is dropping something. Systematically longer usually means it is keeping boilerplate. Both matter; neither shows up as an error.

Structure. Count tables, code blocks and headings in each. A vendor that drops tables will not show up in a text-similarity score at all.

Failures. Which URLs failed for each, whether they were the same URLs, and what a failure looked like on the wire. Different failure sets are informative in both directions.

Cost on your real mix. Not the headline rate. Run your slice and count what each would actually charge including failures, since a vendor that bills for blocked pages costs more than its rate suggests.

Migrate in slices

Move one segment first: one source, one category, one job. Keep everything else on the old vendor.

This preserves attribution. If quality changes, you know whether the migrated slice moved or everything moved, and those have completely different causes. A big-bang cutover throws that away exactly when you need it.

Widen once a slice has run clean through a full cycle, including a re-crawl and at least one site change, which is where caching and freshness differences surface.

Keep the old outputs

Do not delete the previous vendor’s captures when you cut over.

Three months later someone will say results have got worse. Without the old outputs that is unfalsifiable and turns into an argument. With them it is a diff, and the answer arrives in ten minutes: either the extraction changed or the web did.

This is the same argument for keeping raw captures generally, and it applies with particular force during a migration.

Ask the questions that pricing pages do not answer

Before committing, get answers on:

• What happens on a blocked page, exactly, on the wire
• Whether you are billed for it
• Whether a failure is distinguishable from an empty page in the response
• Whether the price multiplies for pages that turn out to be harder to fetch
• What the cache window is, and whether you can control it

That last set decides your real bill more than the per-page rate does. There is more on this in what web scraping actually costs.

The shape of a good migration

1. Pick a representative slice
2. Run both vendors on it, same day
3. Diff content, structure, failures and cost
4. Read the twenty biggest differences by hand
5. Migrate one segment, keep the rest
6. Widen after a full cycle
7. Keep the old outputs

Steps two through four are an afternoon and they are the entire value. Everything else is logistics.

If we are the vendor you are evaluating: every scrape response carries a quality object so failures are explicit rather than inferred, a page costs one credit whatever it took to fetch, and failures are never billed, which means running a parallel comparison costs you only the pages that actually worked. A work email gets you 500 free credits, which is enough for a real slice.