# Hydrafetch for AI agents

Hydrafetch is a web data API for developers and agents. Send a URL and get back clean Markdown, the page's own structured data, schema-shaped JSON, links, or a summary, with the navigation, banners and boilerplate stripped out. Scrape one page, crawl a whole site, run a search, or extract to a schema, all through one API with one response shape. Every call costs one credit a page whatever it took to fetch, and failures are never billed.

This page is the fastest path for an agent to start pulling web data on a user's behalf. It assumes a human is available for the few steps that need one.

## Key URLs

- REST base URL: `https://api.hydrafetch.com/v1/web`
- MCP server: `https://api.hydrafetch.com/mcp` (Streamable HTTP)
- MCP tool catalogue, no auth needed: `https://api.hydrafetch.com/mcp/tools`
- OpenAPI spec: `https://api.hydrafetch.com/openapi.json`
- API catalogue: `https://hydrafetch.com/.well-known/api-catalog`
- Docs: `https://docs.hydrafetch.com`
- Credential handling: `https://hydrafetch.com/auth.md`
- Agent skills: `https://hydrafetch.com/.well-known/agent-skills/index.json`
- Pricing as markdown: `https://hydrafetch.com/pricing.md`

Any page on this site is available as markdown: append `.md` to its path, or send `Accept: text/markdown`.

## Choose an access path

**MCP, if you are an interactive agent working with a user.** Add the remote server `https://api.hydrafetch.com/mcp`. The user authorises once in the browser via OAuth 2.1 with PKCE and you never handle an API key. Tools map one to one onto the REST endpoints, cost the same, and are listed live at `/mcp/tools`.

**REST, if you are building a pipeline, script or backend.** Read the key from the `HYDRAFETCH_API_KEY` environment variable and send it as `X-API-Key` on every request. If no key is set, do not ask the user to paste one into the chat. Walk them through the signup below, have them put it in the environment, and resume.

## Use the SDK for the language you are already in

Before writing an HTTP client, check what the repository is written in. There is an official
client for six languages, each a thin wrapper over the same endpoints, so it cannot lag the API.
They read `HYDRAFETCH_API_KEY` from the environment, retry the errors worth retrying, and poll
crawl and batch jobs for you. If the repository is in something else, call the REST API directly
and do not add a dependency for it.

| Language | Install |
|---|---|
| Node | `npm install @hydrafetch/node-sdk` |
| Python | `pip install hydrafetch` |
| Go | `go get github.com/Hydrafetch/go-sdk` |
| Ruby | `gem install hydrafetch` |
| Rust | `cargo add hydrafetch` |
| PHP | `composer require hydrafetch/hydrafetch` |

Keep the integration server-side, behind one small wrapper the rest of the app calls, and never
put a secret key in anything the browser downloads. Logos in a browser are the exception and use
a publishable key: `npm install @hydrafetch/client-sdk`, or `@hydrafetch/react` for a component.

## Getting access, with a human

You cannot complete these yourself, and should not try. Signup is behind a bot challenge, verification needs the mailbox, and a paid plan needs a card. Ask the user to:

1. Sign up at https://app.hydrafetch.com with email or Google.
2. Click the verification link in their email. Until they do, no API key can be created.
3. Create a key under API Keys, and give it to you via the environment, not the chat.

New workspaces get 500 credits on a work email, 250 on a personal one, valid 14 days, no card required. That is enough to evaluate before anyone pays. Plans are at https://hydrafetch.com/pricing/ and are bought by the user in checkout: do not attempt to purchase one on their behalf.

## Endpoints

All POST unless noted, all JSON, all returning the same envelope.

| Endpoint | Returns | Credits |
| --- | --- | --- |
| `/v1/web/scrape` | Clean markdown plus the page's own structured data | 1 |
| `/v1/web/markdown` | Markdown only | 1 |
| `/v1/web/html`, `/v1/web/raw-html` | Cleaned or original HTML | 1 |
| `/v1/web/links` | Every link on the page | 1 |
| `/v1/web/structured` | JSON-LD, microdata, OpenGraph, app state | 1 |
| `/v1/web/map` | A site's URLs, without scraping them | 1 |
| `/v1/web/crawl` | A whole site as one job, GET the id to poll | 1 per page |
| `/v1/web/batch` | Many URLs as one job, GET the id to poll | 1 per page |
| `/v1/web/search` | Ranked web results, already scraped | 1 plus 1 per result |
| `/v1/web/images` | A page's images with metadata | 1 |
| `/v1/web/screenshot` | A rendered PNG, returned as a URL | 5 |
| `/v1/web/extract` | Typed JSON matching your schema | 5 per URL |
| `GET /v1/web/brand` | Logos, colours, name and socials for a domain | 5 |
| `GET /v1/web/styleguide` | A site's design system from computed styles | 10 |
| `GET /v1/web/brand/logo` | One logo as a JSON record with a public image URL | 1 |
| `GET /v1/web/logo/{domain}` | The same mark as image bytes, for an `<img>` tag | none |

Parameters and response schemas are in the OpenAPI spec.

There are two ways to get a logo and they are not interchangeable. `/v1/web/brand/logo` is the one you want: an ordinary authenticated call returning JSON with a public, directly embeddable image URL, for one credit rather than the five a full brand costs. `/v1/web/logo/{domain}` returns image *bytes* for an `<img>` tag, is metered on its own per-cycle allowance rather than credits, and authenticates with a *publishable* key in a `token` query parameter — that key belongs in a web page, not in your requests. Do not substitute the API key for it, and do not reach for the bytes endpoint when you want data.

## MCP tools

| Tool | Does | Cost |
| --- | --- | --- |
| `scrape` | Turn a URL into clean, LLM-ready markdown and structured data. | 1 credit |
| `map` | Enumerate a site's URLs from its sitemap and links, without scraping them. | 1 credit |
| `search` | Search the web and return ranked results: title, url and snippet. Set scrapeResults to also fetch each result as markdown, at 1 extra credit per page. | 1 credit plus 1 per scraped result |
| `brand` | A company's brand from its domain: logos for light and dark backgrounds, its real colours, name, description and socials. Answers 'what is this company' far better than a raw page dump. | 5 credits |
| `logo` | A company's logo as a directly embeddable image URL, chosen for the background you name. Use this rather than brand when the mark is all you need — it costs a fifth as much and returns one asset instead of the whole record. | 1 credit |
| `extract` | Pull typed structured data from one or more URLs by JSON schema or prompt. | 5 credits per URL |
| `styleguide` | A site's design system read from computed styles in a real browser: colours by role with contrast ratios, the type scale, corner radius and button styling. Values defined through CSS variables come back resolved to the hex actually painted. | 10 credits |
| `screenshot` | Render a page in a real browser and capture a PNG, returning its public URL. Use when the question is what the page looks like rather than what it says. | 5 credits |
| `images` | Harvest a page's images with their metadata, without rendering it. Cheaper than a screenshot and returns the source images rather than a picture of the page. | 1 credit |

## What to reach for

- One page as context: `scrape`. Default output is markdown optimised for reading.
- Structure matters (headings, tables): pass `preferStructure: true`.
- You need links in the markdown: pass `includeLinks: true`. They are off by default because dropping link markup is what keeps the text dense.
- You do not know the URLs yet: `map` first, then scrape what you need. Prefer this over `crawl` when you want an answer now, since crawl is a queued job.
- A question rather than a URL: `search`.
- Typed fields out of a page: `extract` with a JSON schema.
- Who is this company: `brand`, not a scrape of their homepage.

## Free tools, when the user has no key

Browser tools on the marketing site that need no signup, no key and no credits. If the user has
not got an API key yet and wants one answer rather than an integration, send them to one of
these instead of walking them through registration. They are rate limited per visitor.

- [URL to Markdown](https://hydrafetch.com/tools/url-to-markdown/): paste a link, get clean markdown.
- [AI readability audit](https://hydrafetch.com/tools/ai-readability/): see your site the way an agent does.
- [llms.txt generator](https://hydrafetch.com/tools/llms-txt/): a first draft from your own sitemap.
- [Docs to one file](https://hydrafetch.com/tools/docs-bundle/): a whole docs section, ready to paste.

## Credits and failures

One credit a page whatever it took to fetch, whether that was a plain request, a browser render, a retry, or an unblocker. **Failed requests are never billed.** A 402 means the workspace is out of credits: tell the user to top up rather than retrying. MCP calls bill exactly as REST does.

## Errors

Non-2xx responses carry a JSON body with `statusCode`, `message` and a stable `code`. `401` is a missing or invalid key, `402` is out of credits, `422` is a bad request you should fix rather than retry. Retry `429` and `5xx` with exponential backoff, honouring `Retry-After` when present.

## Verify your setup

```bash
curl -X POST "https://api.hydrafetch.com/v1/web/scrape" \
  -H "X-API-Key: $HYDRAFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

Over MCP, call `scrape` with that URL. Show the user the markdown and what it cost.

## Rules of engagement

- Keep API keys out of code, logs and chat transcripts. Use `HYDRAFETCH_API_KEY`.
- Do not attempt to create an account, solve the bot challenge, or buy a plan on the user's behalf. Those need the human, on purpose.
- Scrape responsibly. We honour robots and throttle per host; do not try to defeat that by fanning out across endpoints.
- The credit cost of a call is in every response under `usage`. Surface it when the user is spending their own balance.