What to do when a page comes back empty

“The page came back empty” is one symptom with about six causes, and they need entirely different responses. Guessing wastes hours, and the guess people reach for first (retry it) is usually the least likely to help.

TLDR

• Check the raw response, not the extracted output. That is where the answer is.
• Markup-to-content ratio separates the two main cases: a wall has lots of markup and no content; a thin page has neither.
• A 200 means nothing. Refusals are routinely served as normal pages.
• Retry only transient causes. Retrying a JavaScript-only page forever is a common and pointless loop.
• The dangerous case is the one that is not empty: a wall that extracted cleanly into confident prose.

First, look at the raw response

The single most useful debugging move, and the one most often skipped: stop looking at the extracted output and look at what actually came back.

Three numbers tell you nearly everything:

• Status code
• Raw response size in bytes
• Extracted content length in words

The relationship between the last two is the diagnostic:

raw size	extracted	almost certainly
large	near zero	a wall, or JavaScript-only
small	near zero	genuinely thin, or a failed fetch
large	large	not actually empty; check your reader

The six causes

1. The content needs JavaScript. The HTML is a shell, the content arrives after a script runs. Look for a large document that is mostly <script> tags with an empty root element. Retrying identically will never help.

2. A consent or verification wall. The origin returned a real page, just not the one you wanted. Large markup, tiny content, and phrases about cookies or verifying you are human. Retrying immediately usually makes it worse.

3. The extractor discarded it. The content arrived and your extraction dropped it, which happens on unusual templates where the main content is not in the shape the extractor expects. Distinguishing feature: the raw response clearly contains the article. This is a reader problem, not a fetch problem.

4. The page is genuinely thin. Some pages really are a heading and a form. Nothing is wrong, and the correct handling is to record it as thin rather than as a failure.

5. A redirect took you elsewhere. You asked for an article and landed on a homepage, a login, or a region selector. The final URL differing from the requested one is the giveaway, which is why it is worth recording both.

6. The fetch failed but returned a body. Error pages are pages. A 404 or a 503 often has a nicely styled document that extracts to almost nothing.

The case that should worry you more

Everything above is visible. The expensive failure is the one that does not look like a failure at all.

A consent wall is prose. Extracted, it becomes clean, well-formed markdown that says something about cookies and your choices. It has a reasonable word count. It passes an “is it empty” check comfortably, because it is not empty.

It then embeds cleanly, indexes cleanly, and sits in your system until it produces a confident answer built on a cookie notice.

So “did we get content” is the wrong question. The right one is “did we get the content we asked for”, and answering it needs more than a length check. A quality signal on the response, a blocked flag, or at minimum a check for wall-shaped phrasing on suspiciously short results.

What to do in each case

cause	response
Needs JavaScript	render it, or accept it will not work statically
Wall	back off, do not hammer; treat as a failure, not content
Extractor dropped it	fix the reader; the bytes were there
Genuinely thin	record as thin, do not retry
Redirect	follow and record the final URL, or treat as a miss
Failed fetch	retry with backoff if the status warrants it

The important column is the last one. Only two of the six are worth retrying, and both need backoff. A pipeline that retries everything three times spends four times the requests to get the same answer, and looks considerably worse to the origin while doing it.

Record enough to diagnose later

Most of the difficulty in debugging this is that the evidence is gone by the time anyone looks. Keep, per fetch:

• The requested URL and the final URL
• The status code
• The raw response size
• The extracted word count
• Whatever quality or blocked signal you have

Five fields, all small, and they turn “this page is empty sometimes” from an investigation into a query. Storing the raw capture as well means you can re-extract when you eventually improve the reader, rather than refetching a page that may have changed.

The one-minute triage

1. Is the raw response large? If yes, the fetch worked and it is a wall, JavaScript, or your extractor.
2. Does it mention cookies or verification? Wall.
3. Is it mostly script tags with an empty root? JavaScript.
4. Can you see the article in the raw HTML? Your extractor.
5. Is the raw response small too? Genuinely thin, a redirect, or a failed fetch. Check the final URL and status.

Nearly every case resolves in under a minute with the raw response in front of you, and takes an afternoon without it.

Every scrape response carries the final URL, the status and a quality object with a confidence score, a completeness flag and a blocked flag, so your code can branch on all of this instead of guessing from length. Pages that came back walled are never billed. A work email gets you 500 free credits to try it against whichever URL is currently returning nothing.