Structured data extraction you can trust

Turning pages into records is one of those tasks that looks solved until you run it for a month. Selectors break on a redesign you did not know happened. A model fills every field you asked for, including the ones the page never mentioned. This post covers both failure modes, and how to get output you can check rather than hope about.

TLDR

• Selectors encode markup, not meaning, so they break on redesigns and usually fail silently by returning nothing.
• Models given raw HTML fill required fields whether or not the page supports them. Well-formed and wrong is the harder problem.
• Ask for evidence. Every field can carry the passage it came from and a confidence figure.
• Check whether the page already publishes structured data. If it does, that costs one credit and no model runs.
• Schema extraction is five credits per URL. Pages that cannot be read cost nothing.

Why do CSS selectors keep breaking?

Because a selector describes where something sits in the markup, and the markup is not a contract. div.product-grid > div:nth-child(2) .price is a claim about layout that any redesign, component rename or A/B test can invalidate without anyone telling you.

The failure mode is what makes it expensive. A broken selector does not throw. It returns nothing, or it returns the wrong element, and your pipeline records a null or a plausible-looking mistake. You find out when someone downstream asks why the numbers moved, and by then you have days of bad rows.

Multiply by the number of sites you cover and selector maintenance becomes the job. That is the work most teams are actually trying to stop doing.

Why not just hand the HTML to a model?

Because a model will answer the question you asked even when the page does not. Given a schema with a required price field and a page that does not list a price, the likeliest output is a number. It will be well-formed, it will match your type, and it will be wrong.

This is worse than a broken selector in one specific way: a null is obviously missing, and a confident fabrication is not. Validation catches shape, not truth.

There is a cheaper problem too. Raw HTML is mostly markup, so you pay model tokens to process navigation and scripts. A Wikipedia article is 59,769 tokens as HTML and 6,831 as clean markdown. Extracting from the second costs a fraction of the first, before you have improved accuracy at all.

How does evidence change this?

It moves you from trusting output to checking it. Ask for sources and confidence, and each field arrives with the passage it was drawn from.

curl -X POST https://api.hydrafetch.com/v1/web/extract \
  -H "X-API-Key: $HYDRAFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing"],
    "schema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "monthlyUsd": { "type": "number" }
            }
          }
        }
      }
    },
    "showSources": true,
    "showConfidence": true
  }'

Now you can gate on evidence rather than accept the whole record:

const { data } = await res.json();

const trusted = data.json.plans.filter((plan, i) => {
  const field = data.sources?.[`plans[${i}].monthlyUsd`];
  return field && field.confidence > 0.8;
});

That filter is the difference between a pipeline that degrades loudly and one that degrades silently. Low-confidence rows can go to a review queue, or be dropped, or trigger a re-fetch. What they should not do is land in your database looking identical to rows that were actually on the page.

When do you not need a model at all?

More often than you would expect. A large share of the commercial web already publishes machine-readable data in the markup: JSON-LD, microdata, RDFa, Open Graph. Product pages, articles, recipes, events and job postings are frequently annotated because search engines reward it.

If a page publishes its own structured data, a plain scrape returns it. That costs one credit, involves no model, and cannot hallucinate, because nothing is being inferred.

Situation	Reach for	Cost
Page publishes JSON-LD or microdata	structured format on scrape	1 credit
Page has the facts in prose only	Schema extraction	5 credits
Same entity across several pages	Extraction with entity merging	5 credits per URL
Logos, palette, fonts for a company	Brand endpoint	5 credits

Checking the cheap path first is worth building into your pipeline rather than deciding once. On a catalog of mixed sources, the split between the two is usually large enough to notice on the bill.

How do you keep this honest over time?

The same way you keep extraction honest: measure it, and keep the evidence.

Extraction quality underneath all of this is measurable. Our engine scores 0.859 word-level F1 on the held-out split of a 2,008 page human-labeled corpus, against 0.857 for Trafilatura, the strong open-source baseline, on the same pages. That matters here because schema extraction runs on top of the cleaned text. Extraction from a page whose main content was half dropped produces confident answers about the half that survived.

Two habits worth adopting:

• Store the evidence, not just the value. When a number is disputed six months later, the passage is the answer.
• Sample by hand. Fifty rows read properly tell you more than any aggregate confidence score, and they are how you discover that one source has been quietly failing.

We covered how to measure cleanliness itself in what clean web data for LLMs actually means, and the structured extraction use case page has the compressed version of this.

Where to start

Pick the source that breaks most often. Run one page through schema extraction with showSources and showConfidence on, and read the evidence next to the values. If the passages support the values, you have a pipeline you can check. If they do not, you have just found out cheaply, which is the whole point.