Using search results as a data source

Search is the only tool that works when you do not know which page holds the answer. It is also, of everything you might feed a pipeline, the least reproducible. Both facts matter, and the second is the one that surprises people.

TLDR

• Use search when the URL is part of the question. If you know the URL, fetch it.
• Results are not reproducible: rankings shift, and personalisation and region change them.
• Snippets are advertising, not content. Answering from them is a reliable way to be confidently wrong.
• Fetch the top few, not ten. The answer is usually in the first three.
• Check that operators were honoured, because a silently ignored site: looks exactly like a good result.

Search is for discovery, not retrieval

The distinction is worth being strict about, because conflating them is expensive.

Retrieval is “get me this page”. You know the URL. Fetch it directly: cheap, fast, and it returns the same thing every time.

Discovery is “which page answers this”. You do not know the URL, and finding it is the actual problem.

Agents frequently search when they already know the answer’s location, because searching feels like the general-purpose move. It costs a round trip, introduces ranking noise, and can return something worse than the page you already had.

Nothing about search is stable

If you have only used search as a human, its instability is easy to underestimate.

Rankings move. The same query today and next week returns a different order, sometimes a different set. Nothing announces the change.

Results are personalised and regionalised. Where the request appears to come from changes what comes back. Comparisons across time are only meaningful if that stayed constant.

Indexes differ. Different engines have genuinely different coverage. A page that ranks well in one may be absent from another entirely.

So search results are evidence, not a source of record. If you need something reproducible, resolve the URL once and store it. Re-searching to find the same page again is how a pipeline quietly changes what it reads.

Snippets are not content

The most common mistake, and the most damaging: answering from the snippets.

A snippet is a fragment chosen to look relevant to the query. That is its job. It frequently omits the qualifier that changes the meaning, and it is sometimes assembled from disconnected parts of the page.

A system that reads snippets and answers produces text that looks well-sourced and cites pages it never read. The fix is not subtle: fetch the pages. Search tells you where to look; the page tells you what it says.

Fetch fewer results than you think

The instinct is to grab ten results for safety. In practice the answer is in the first three most of the time, and the rest is cost.

Ten results means ten fetches, ten extractions and ten times the tokens if they all reach the model. It also degrades the answer, because a model given ten documents where two are relevant has to work out which, and it does not always get that right.

Start with three. Fetch more only when the first three genuinely do not answer it, which is a decision you can make after reading them.

Use operators, then check they worked

Operators are how you narrow before spending anything:

• site: pins a domain, which is the highest-value one by far
• -term excludes
• Quotes force an exact phrase

The trap: not every engine honours them. Some silently drop site: and answer the general query instead, returning confident, well-formed results about the wrong subject. Nothing signals this, and the results look fine.

So verify. If you asked for site:example.com and a result is not on that domain, the operator was ignored and the whole result set is suspect. This check costs nothing and catches a failure that is otherwise invisible.

Distinguish “no results” from “could not search”

The failure mode that matters in production. Two very different situations look identical if you are careless:

• The web genuinely has no match for the query
• Search itself failed and returned nothing

An empty list means the first. An error means the second. If your pipeline collapses both into “no results”, then a search outage looks like the world having no answer, and your system will confidently report absence for hours.

This is worth checking in whatever you build on, ours included: ask what an outage looks like on the wire, and make sure it is not an empty success.

A shape that works

1. Search with operators, few results
2. Verify the operators were honoured
3. Fetch the top two or three pages properly
4. Answer from the page content, citing which page
5. Store the resolved URL, so next time is retrieval rather than discovery

Step five compounds. Most questions get asked more than once, and the second time you should not be searching at all.

Our search endpoint returns ranked results for 1 credit and honours site:. Setting scrapeResults fetches each result as clean markdown for 1 credit a page, so you can go from question to page content in one call, and a total search failure returns an error rather than an empty list. A work email gets you 500 free credits.