Ground a RAG index on content, not chrome.

Crawl a whole site on a schedule and pipe clean Markdown straight into your embeddings.

What it costs

Per page crawled: 1 credit
Pages that failed: Free
Re-processing a stored page: No refetch

[ The problem ]

Boilerplate is the quiet tax on retrieval

Point a splitter at raw HTML and the nav bar, the cookie notice and the footer become chunks like any other. They embed, they get retrieved, and they crowd out the passage that actually answered the question.

Every repeated element is embedded once per page, so a 300-page site pays for its footer 300 times
Retrieval surfaces chrome that is textually similar to everything else on the site
You pay to embed and to store text no one will ever want returned

Step 01

Start a crawl. It runs as an asynchronous job with depth and path controls, and calls your webhook when it finishes.

crawl.sh

curl -X POST https://api.hydrafetch.com/v1/web/crawl \
  -H "X-API-Key: $HYDRAFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 500,
    "maxDepth": 3,
    "excludePaths": ["/changelog/*"],
    "webhook": { "url": "https://yourapp.com/hooks/hydrafetch" }
  }'

Step 02

Every page arrives in the same envelope. Take the Markdown, keep the metadata as chunk attributes, and skip anything the response flagged as thin.

ingest.py

for page in pages:
    data = page["data"]
    if data["quality"]["confidence"] < 0.5:
        continue

    chunks = splitter.split_text(data["markdown"])
    index.upsert([
        {
            "text": chunk,
            "source": data["finalUrl"],
            "title": data["metadata"].get("title"),
        }
        for chunk in chunks
    ])

[ Worth knowing ]

Every response carries a confidence figure, so a page that came back thin can be skipped rather than silently poisoning the index.

Raw HTML is stored immutably, so changing your chunking strategy later does not mean paying to crawl the site again.

Other use cases

Agents

Search mid-conversation and get results that are already scraped, not a list of links to go and fetch.

Structured extraction

A JSON schema in, typed records out, and the passage each value was drawn from.

Brand data

Logos that work on light and dark, the real colours ranked by how the site uses them, fonts and socials.

[ Start ]

Clean web data is one call away.

250 free credits, no card required. Failures are never billed.

Get API key Read the docs