Ten mistakes people make building on web data

We have written a lot this year about specific parts of working with web data. This is the summary: the mistakes that come up over and over, roughly in order of how much they cost, and where to read more on each.

Every one is invisible in aggregate metrics and obvious when you read twenty outputs by hand. That is the theme.

1. Trusting the status code

A 200 means the server sent a document. It does not mean it sent the document you asked for. Consent walls, verification pages and region gates all return 200 with well-formed HTML.

A pipeline that branches on status code alone ingests all of them as content. This is the single most common defect and the most expensive, because the output looks perfect.

Instead: check the content, not the code. Word count, a wall-shaped-phrasing check, or a quality signal on the response. More in what to do when a page comes back empty.

2. Ingesting the boilerplate

Every page of a site carries the same navigation, footer and cookie banner. Ingest raw HTML and all of it goes into your index, on every page.

You pay for it twice, at ingestion and on every retrieval, and it makes unrelated pages look similar to each other, which quietly degrades retrieval in a way no amount of reranking fixes.

Instead: extract the main content first. See what a web page costs in tokens.

3. Flattening structure

Stripping to plain text is easy and throws away the things that carry meaning. Tables become runs of numbers, code blocks lose their boundaries, headings stop marking sections.

Instead: keep markdown structure. It is what makes structural chunking possible, which is covered in how to chunk web pages for retrieval.

4. Fixed-size chunking

Splitting every N tokens cuts through the middle of explanations and separates a heading from what it introduces. The chunks embed fine. The answers get worse and the cause is nowhere near the retriever.

Instead: split on headings, then paragraphs, and never through a table or code block.

5. Silently dropping failures

A dataset containing only successes looks complete, and every consumer treats it as complete. The pages you could not fetch are not a random sample, so what you have is a biased view of an easier world.

Instead: record failures as rows with reasons. See how to build a dataset from the web.

6. Not keeping the raw capture

Your schema will change and your extraction will improve. Without stored captures, both mean re-crawling everything, weeks later, when the pages have moved on.

Instead: store the raw page alongside the extracted record. It compresses well and it is cheaper than the alternative every single time.

7. Crawling impolitely

Unlimited concurrency against one host is indistinguishable from an attack, and the response is the same. Most blocked crawlers were blocked for rudeness rather than sophistication.

Instead: one host at a time, back off on 429 and 5xx, use conditional requests. See how to crawl a site without being a problem.

8. Refetching what has not changed

Re-crawling everything nightly, when the site changes weekly, means most of your requests and most of your bill produce byte-identical results.

Instead: conditional requests and sitemap lastmod. Most re-crawls can skip the large majority of pages.

9. Answering from snippets

Search snippets are chosen to look relevant. They routinely omit the qualifier that changes the meaning. Systems that answer from them cite pages they never read.

Instead: search to find the page, then fetch the page. See using search results as a data source.

10. Filling a field because the schema has one

The enrichment version of the same disease. A schema with a headcount field creates pressure to produce a headcount, and a plausible guess written into a structured field is indistinguishable from a real value forever.

Instead: allow nulls and mean them. An empty field is a known unknown. See enriching company records from their websites.

The thread running through all of them

Every mistake here shares a shape: the failure produces output that looks correct.

Nothing throws. No error rate moves. The document is well-formed, the chunks embed, the field is populated. That is why aggregate monitoring does not catch any of it, and why the most valuable hour you can spend on a web data pipeline is reading twenty of its outputs end to end.

If you want the parts of this that are ours to solve handled: scrape returns clean markdown with structure intact and a quality signal on every response, so a wall or a truncated capture is something your code can branch on rather than something you find out about later. One credit a page whatever it took to fetch, failures never billed, and a work email gets you 500 free credits.

You can also try a single URL with no signup in the URL to markdown tool.