How to build a dataset from the web

Building a dataset from the web sounds like a collection problem and is mostly a bookkeeping problem. Fetching pages is a solved thing. Knowing what you actually have at the end, and what you are missing, is where these projects succeed or quietly fail.

TLDR

• Define the record first. One page is not automatically one row.
• Record failures as rows, with reasons. Silent omission is how a dataset becomes biased without anyone noticing.
• Keep the raw captures. Extraction will improve and your schema will change, and refetching is expensive and rude.
• Store URL and timestamp per record, or nothing is reproducible.
• The most common flaw is survivorship: the pages that were hard to fetch are missing, and they are not a random sample.

Define the record before you collect anything

The first question is not which sites, it is what one row represents. It sounds procedural and it determines everything downstream.

Is a record a page? A product? A company? A product’s price at a point in time? These produce different pipelines:

• One page, one record is simplest, and wrong whenever a page lists many things or one thing spans several pages.
• One entity, many pages means resolving which pages describe the same thing, which is its own problem.
• One observation means the same entity produces a new row each time you look, and time becomes part of the key.

Deciding late is expensive, because it changes your storage, your deduplication and your idea of what completeness even means.

Discovery is a separate step from collection

Work out which URLs should exist before fetching any of them. Sitemaps, an index page, an existing list you already hold.

Doing this separately gives you something valuable: a denominator. You know you expected 4,000 pages, which means at the end you can say you got 3,760 and where the other 240 went. Discover and fetch in one pass and you only ever know how many you got, which sounds the same and is not.

Record what failed, in the dataset

The most important habit here, and the most often skipped.

When a page fails, the tempting thing is to log it and move on, leaving the dataset with only successes. That dataset now looks complete. Every downstream consumer will treat it as complete.

Instead, write a row:

{"url": "...", "status": "ok", "data": {...}, "fetched_at": "..."}
{"url": "...", "status": "blocked", "data": null, "fetched_at": "..."}
{"url": "...", "status": "not_found", "data": null, "fetched_at": "..."}
{"url": "...", "status": "thin", "data": null, "fetched_at": "..."}

Now the shape of the gap is visible. If every failure is from one domain, that is a finding about that domain, not missing data. If failures cluster in one category, any conclusion you draw from the dataset is biased in a direction you can now name.

This is the difference between a dataset with 94% coverage and a dataset that appears to have 100% coverage of a different, smaller population.

Keep the raw captures

Store the raw page alongside the extracted record. It is the single decision people most regret skipping.

Your schema will change. You will want a field you did not extract. With raw captures that is a re-parse; without them it is a re-crawl of every page, weeks later, when many have changed.

Your extraction will improve. Better parsing applied to captures you already hold costs nothing and changes nothing about the sites you took them from.

Disputes need evidence. When a value looks wrong, the page as it was at fetch time settles it in seconds.

Raw HTML compresses well. This is cheap relative to what it saves.

Make each record reproducible

Three fields on every row, no exceptions:

• The URL, exactly as fetched, including the final URL after redirects
• The timestamp of the fetch
• A pointer to the raw capture

Without these, a wrong value is unarguable: nobody can tell whether the extraction was wrong or the page has simply changed since. With them, every value is traceable to a specific document at a specific moment, which is what makes a web-derived dataset defensible rather than merely available.

Watch for survivorship

The bias that gets into almost every web dataset, and it is worth stating plainly: the pages you could not fetch are not a random sample.

Harder-to-fetch sites skew toward larger companies with more aggressive protection, or toward older sites with unusual markup, or toward whichever platform your extractor handles worst. Drop them silently and your dataset over-represents the easy half of the world.

Failure rows make this checkable. Look at the failures by domain, by category, by whatever dimension you care about, and ask whether their absence changes the conclusion. Sometimes it does not. Sometimes it is the finding.

Refresh deliberately, or not at all

Decide up front whether this is a snapshot or a living dataset, because they are different products.

A snapshot is dated and never changes. It is reproducible by construction and it goes stale by design, which is fine as long as everyone knows the date.

A living dataset needs a refresh policy, per-record timestamps, and a decision about whether history is kept or overwritten. Overwriting is simpler and destroys the ability to answer “when did this change”, which is often the most interesting question in the data.

Choosing accidentally is what produces a dataset where some rows are two years old and nothing records which.

The checklist

1. Define what one record is
2. Discover the URL list, and keep it as your denominator
3. Fetch politely, one host at a time
4. Write rows for failures with reasons
5. Store URL, timestamp and raw capture per record
6. Check the failures for bias before drawing conclusions

Four of those six are bookkeeping. That is the honest ratio of this work.

If you want the collection half handled, map gives you the URL list for 1 credit, crawl and batch fetch them at one credit a page with per-page status so failures arrive as data rather than as gaps, and failures are never billed. A work email gets you 500 free credits to build a real sample and see what your failure rows look like.