What clean web data for LLMs actually means

Every web data API says its output is clean. Almost none of them say what that means, and fewer still measure it. This post pins the word down: what clean actually is, why it decides whether your LLM pipeline works, and how to check any vendor’s claim, including ours.

TLDR

• Raw HTML is mostly noise. A real Wikipedia article is 59,769 tokens as HTML and 6,831 tokens as clean markdown. Same content, 89 percent fewer tokens.
• Clean means three things at once: only the main content, with its structure preserved, and nothing missing.
• Cleanliness is measurable. The standard is word-level F1 against human-labeled pages. Our engine scores 0.857 on a held-out split of a 2,008 page corpus.
• A clean response should also tell you how clean it is. Every response we return carries a confidence score, a completeness flag, and a blocked flag.
• Failures should be free. If the content did not come back clean, you should not be billed for it.

Why does raw HTML hurt LLM pipelines?

Raw HTML hurts because models pay for every token and trust what they read. A typical content page is 80 to 90 percent markup, scripts, navigation, ads and boilerplate. Feed that to a model and you pay to process noise, and the noise competes with the signal.

The numbers are not subtle. The Wikipedia article on web scraping is 239,076 characters of raw HTML, which tokenizes to 59,769 tokens. The clean markdown of the same article is 27,326 characters and 6,831 tokens. That is the same information at 11 percent of the cost, before you multiply by every page in your index and every model call that reads it.

Cost is the visible half. The quieter problem is retrieval quality. If your RAG index embeds cookie banners and footer sludge, those chunks match queries they should never match. Cleaning the input is the cheapest quality improvement most pipelines can make.

What does “clean” actually mean?

Clean means the main content, the whole main content, and nothing else. It helps to split that into three tests, because vendors routinely pass one and fail the others.

Test	Question it answers	Common failure
Boilerplate removal	Did the nav, ads and banners go away?	Footer links survive into the output
Structure preservation	Do headings, lists and tables still exist?	Everything flattens into one wall of text
Completeness	Did the whole article make it out?	Extractor stops at the first comment block

Boilerplate removal is what people picture when they hear clean. Structure preservation is what makes the output usable: a model reasons better over ## Pricing followed by a table than over the same words run together. Completeness is the one nobody demos, because its failure mode is silent. An extraction that drops the last half of the article looks perfectly clean.

How do you measure extraction quality?

Word-level F1 against human labels. Humans mark which words on a page are main content. The extractor runs on the same page. Precision asks how much of what it kept was right, recall asks how much of the right content it kept, and F1 combines the two. It punishes both kinds of failure: keeping junk and dropping content.

Our engine scores 0.857 word-level F1 on a held-out split of a 2,008 page human-labeled corpus. For calibration, Trafilatura, the strongest open-source extractor and the baseline worth beating, scores 0.841 on the same set. The suite re-runs against the live deployed engine before any extraction change ships, so the number on the site is the number in production.

If a vendor claims clean output, ask for the F1 and ask for the denominator. Accuracy without a denominator is a slogan.

What should a quality signal look like?

Every response should tell you how much to trust it, without costing extra. Ours carries a quality object on every scrape:

"quality": { "confidence": 0.99, "complete": true, "blocked": false }

Confidence is how sure the engine is about the extraction, from 0 to 1. Complete says whether the bulk of the page made it out. Blocked says whether a challenge wall stood in the way of the real content.

These flags exist because the worst outcome in web data is not a failed request. It is a 200 response containing a CAPTCHA page that your pipeline happily embeds. A blocked page should be flagged as blocked, and it should never be billed. That is also how our pricing works: a page that did not come back clean costs zero credits.

What about JavaScript-rendered pages?

They should come back as content, and you should not have to do anything to make that happen. A large share of the modern web only exists after the browser draws it. If your extractor sees an empty <div id="root">, the cleanest markdown in the world is still empty.

We render when the page needs it, automatically, and the result costs the same one credit as any other page. If a vendor makes you choose render modes per URL, they have exported their problem to you: you cannot know in advance which pages need it, and guessing wrong gives you empty shells in your index.

How to evaluate any vendor in an afternoon

Take ten URLs you actually care about, not demo pages. Include at least one JavaScript-heavy app, one news page with a paywall or cookie wall, and one long article.

1. Fetch each URL through the API and count the output tokens against the raw page.
2. Read the output for the long article and check the ending made it.
3. Look for a quality signal on the response. If there is none, ask how you are supposed to catch silent failures.
4. Check the bill for the pages that failed.

That hour tells you more than any benchmark table, ours included. If you want the starting point, the scrape endpoint returns clean markdown with the quality object on every response, and a work email gets you 500 free credits to run exactly this test.