How to ground an AI agent in the live web

An agent that cannot read the live web is reasoning from whatever was in its training data. The usual fix is to bolt on a search API, and it half works: the agent gets a list of links and then has to go and fetch them, which is where most of the difficulty actually lives. This post covers what grounding needs, and the two ways to wire it up.

TLDR

• A list of links is not grounding. Snippets are written for humans scanning a results page, not for a model that needs the passage.
• One call can search and scrape together, so results arrive as markdown the agent can quote.
• Over MCP the agent calls the tools itself, so you stop writing orchestration for when to search and when to read.
• Keep the source URL attached to the text at every hop, or citations become guesses.
• A search costs one credit plus one for each result scraped. Failed calls cost nothing.

Why is a list of links not enough?

Because the agent still has to do the hard part, and it is not equipped to. A search API returns a title, a URL and a snippet of a couple of sentences. That snippet was chosen to help a person decide whether to click, not to contain the answer.

So the agent fetches the page itself, and meets everything that makes fetching pages hard: pages that only exist after JavaScript runs, sites that refuse automated clients, and HTML where the article is 10 percent of the bytes. Now that logic lives in your agent code, and every failure mode shows up as the agent saying something vague.

The alternative is to make retrieval return content rather than pointers.

How does search-then-scrape work in one call?

You send a query and ask for the results to be scraped. Each result comes back ranked and already reduced to markdown.

curl -X POST https://api.hydrafetch.com/v1/web/search \
  -H "X-API-Key: $HYDRAFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "retrieval augmented generation evaluation",
    "limit": 5,
    "scrapeResults": true
  }'

The agent now has five pages of actual content instead of five links and a plan. Set scrapeResults to false when the agent only needs to know what exists, for example when it is checking whether a topic has coverage before committing to a deeper read, and you pay for the search alone.

Leanness matters more here than in a batch pipeline, because every token competes with the rest of the conversation. Against Jina Reader across ten URLs through the full pipeline, our output was 64 percent leaner, 25k tokens against 69k, on the same pages. In a batch job that is a cost line. In an agent loop it is the difference between five pages fitting in context and two.

What does connecting over MCP change?

It moves the decision from your code to the agent. MCP is an open protocol for exposing tools to a model, and connecting an agent to our endpoint gives it scrape, map, search, extract, brand and usage as tools it can call whenever it decides it needs them.

claude mcp add --scope user --transport http hydrafetch https://api.hydrafetch.com/mcp \
  --header "Authorization: Bearer $HYDRAFETCH_API_KEY"

That is the whole integration. No SDK, nothing running locally, and the same for Cursor, Codex, OpenCode, Cline and Windsurf with their own config shapes, which are on the integrations page.

The practical difference: without MCP you write the rules for when to search, when to read a full page, and when to stop. With MCP the agent asks for what it needs, and you spend your time on the prompt instead of the plumbing. A tool call costs the same as the matching HTTP endpoint, so this is not a more expensive path.

How do I keep citations honest?

Carry the URL with the text, every hop, and make the agent quote from what it was given.

Grounding fails in a specific way that is easy to miss in testing. The model reads five pages, synthesises an answer, and attaches a citation that is plausible but points at the wrong one of the five. It looks correct unless you check, and nobody checks every answer.

Three habits prevent most of it:

• Keep finalUrl next to the markdown in whatever structure you pass to the model, so the association survives redirects.
• Ask the model to quote the sentence it is relying on, not just cite the page. A quote can be verified against the text you hold; a citation cannot.
• Give it fewer, better pages. Five clean pages beat twenty noisy ones, because the noise is what it misattributes.

When should an agent not go to the web?

When you already have the answer, which is more often than it feels. Live retrieval costs latency and credits on every call, and an agent that searches reflexively is slow and expensive without being more correct.

Situation	Better move
Stable reference material	Index it once, retrieve locally
The same query across many sessions	Cache the result and set a freshness bound
Facts that change hourly	Go live, every time
Whole site needed, not one page	Crawl it into an index instead

The last row is the one people get wrong most. If an agent keeps reading pages from the same domain, that domain belongs in a retrieval index, not in the agent loop. We covered that pipeline in web data for RAG.

What does it cost?

A search is one credit, and each result it scrapes is one more. A five result search with scraping is six credits. A page read on its own is one credit, whatever that page took to deliver, and a call that fails costs nothing at all.

That last point matters more for agents than for batch jobs. Agents retry, and they wander into pages that cannot be delivered. If failures were billed, the least predictable part of your system would also be the most expensive. The full table is on pricing, and the shorter version of this setup is on the agents use case page.

Where to start

Take the question your agent gets wrong most often, and give it one search-then-scrape call before it answers. Compare the two answers side by side. If the grounded one is better, the next decision is whether you want to keep writing the orchestration or hand the agent the tools and let it decide.