---
title: "Hydrafetch — Clean web data for developers and agents"
url: https://hydrafetch.com/?utm_source=akashrajpurohit.com&utm_medium=consulting
description: "Turn any URL into clean Markdown and structured data your model can use. Scrape, crawl, search and extract with one API, one credit a page."
---
[hydrafetch](https://hydrafetch.com/)[Book a call](https://hydrafetch.com/demo/)[Get started](https://app.hydrafetch.com/)
Endpoints
[ScrapeClean markdown from one URL](https://docs.hydrafetch.com/endpoints/scrape)[MapEvery URL on a site, one call](https://docs.hydrafetch.com/endpoints/map)[CrawlWhole sites as one job](https://docs.hydrafetch.com/endpoints/crawl)[SearchRanked results, already fetched](https://docs.hydrafetch.com/endpoints/search)[ExtractTyped records from a schema](https://docs.hydrafetch.com/endpoints/extract)[BrandA company from its domain](https://docs.hydrafetch.com/endpoints/brand)
Concepts
[FormatsMarkdown, structured, links](https://docs.hydrafetch.com/concepts/formats)[QualityConfidence on every response](https://docs.hydrafetch.com/concepts/quality)[CreditsOne a page, failures free](https://docs.hydrafetch.com/concepts/credits)[CachingFresh or cached, your call](https://docs.hydrafetch.com/concepts/caching)
[Try it Run any endpoint on a URL of your own, no key](https://hydrafetch.com/try/)
Web data for AI
[Ground RAG in fresh contentCrawl a site on a schedule and pipe clean Markdown into your embeddings.](https://hydrafetch.com/use-cases/rag/)[Give an agent the live webSearch mid-answer and get results already fetched, cleaned and citable.](https://hydrafetch.com/use-cases/agents/)[Turn listings into a datasetPoint a JSON schema at a directory and get typed rows back.](https://hydrafetch.com/use-cases/structured-extraction/)[Watch pages for changesRe-run a set on a schedule and diff what came back.](https://hydrafetch.com/use-cases/change-monitoring/)[Track prices and competitorsMap a catalogue, pull every page, and extract the fields that move.](https://hydrafetch.com/use-cases/competitor-intelligence/)
Company and brand data
[Enrich a company from its domainOne call returns the name, industry, assets, palette and socials.](https://hydrafetch.com/use-cases/company-enrichment/)[Theme your app per tenantResolve a domain to a full design system and restyle your UI at runtime.](https://hydrafetch.com/use-cases/white-label-theming/)[Autofill onboardingTurn a work email domain into a filled-in company form.](https://hydrafetch.com/use-cases/onboarding-autofill/)
[All use cases Each one worked end to end, on real captures](https://hydrafetch.com/use-cases/)
[DocsEndpoints, formats and limits](https://docs.hydrafetch.com/)[BlogHow the engine is measured](https://hydrafetch.com/blog/)[Extraction benchmarkWord-level F1 against the field](https://hydrafetch.com/reports/extraction-benchmark/)[Token indexWhat a page costs once it is clean](https://hydrafetch.com/reports/token-index/)
[Talk to a human Thirty minutes, and bring your hardest URL](https://hydrafetch.com/demo/)
# Clean web data for developers and agents
Any URL to markdown or structured data. Scrape, crawl, search and extract from one API, one credit a page.
[Get API key](https://app.hydrafetch.com/)
https://
TryNo signup needed[All endpoints ↗](https://hydrafetch.com/try/)
[01 / Endpoints]
## Every endpoint. One response shape.
From one page to a whole site, the envelope never changes. Learn one and you have learned all of them.
One URL in. Clean Markdown, the page’s own structured data, links, or a summary out, each response carrying its quality score. The primitive everything else builds on.
[Read the docs](https://docs.hydrafetch.com/endpoints/scrape)
Every URL on a site in one call, sitemap and discovery combined, deduplicated and orderable by relevance. See the whole site before you decide what is worth pulling.
[Read the docs](https://docs.hydrafetch.com/endpoints/map)
Walk a whole site as one asynchronous job, with depth, path and subdomain controls and a webhook when it finishes. Every page returns as clean Markdown in the same envelope.
[Read the docs](https://docs.hydrafetch.com/endpoints/crawl)
Run a query and get ranked results that arrive already fetched and cleaned, with time, region and domain filters. Fresh, cited context for a model in one round trip.
[Read the docs](https://docs.hydrafetch.com/endpoints/search)
Point a JSON schema at one page or a whole site and get typed records back, deduplicated across pages, with per-field confidence and the passage each value was drawn from.
[Read the docs](https://docs.hydrafetch.com/endpoints/extract)
Give it a domain and get the company ready to render: logos that work on light and dark, the brand’s real colours ranked by how the site uses them, fonts and socials. Every field carries its source and confidence.
[Read the docs](https://docs.hydrafetch.com/api-reference/brand/resolve-a-brand)
The brand engine’s visible tip. One URL that drops straight into an img tag and returns the company’s mark, correct on light or dark, authenticated by a key that is safe to publish in your markup.
[Read the docs](https://docs.hydrafetch.com/endpoints/logo)
{"url": "en.wikipedia.org/wiki/Web_scraping"}
```
**Published:** 2005-09-17
# Web scraping
**Web scraping**, **web harvesting**, or **web data extraction** is data scraping used for extracting data from websites.
Scraping a web page involves fetching it and then extracting data from it. Fetching is the downloading of a page.
Web pages are built using text-based markup languages (HTML and XHTML), and frequently contain useful data in text form.
## History
After the birth of the World Wide Web in 1989, the first web robot was created in June 1993.
In December 1993, the first crawler-based web search engine, JumpStation, was launched.
## Techniques
Data extraction techniques range from manual collection to sophisticated automation.
### Human copy-and-paste
```
59,769 → 6,831 tokensconfidence 0.991 credit
{"url": "docs.hydrafetch.com"}
docs.hydrafetch.com
├── api-reference/
│ ├── brand/
│ │ ├── get-a-company-logo2026-08-22
│ │ ├── read-a-companys-design-system2026-08-22
│ │ ├── resolve-many-brands2026-08-22
│ │ └── search-brands2026-08-22
│ ├── crawl-&-batch/
│ │ ├── get-crawl-status2026-08-22
│ │ ├── map-a-site2026-08-22
│ │ └── start-a-crawl2026-08-22
│ ├── extract/
│ │ └── extract-structured-data-from-urls2026-08-22
│ ├── logo/
│ │ └── serve-a-company-logo2026-08-22
│ ├── media/
│ │ ├── harvest-a-pages-images2026-08-22
│ │ └── screenshot-a-page2026-08-22
│ ├── scrape/
│ │ ├── get-a-pages-links2026-08-22
│ │ ├── get-scrape-job-status2026-08-22
│ │ ├── scrape-a-url2026-08-22
│ │ └── scrape-to-markdown2026-08-22
│ └── search/
│ └── search-the-web2026-08-22
├── changelog2026-08-22
├── concepts/
│ ├── caching2026-08-13
│ ├── errors2026-08-14
│ ├── formats2026-08-22
│ ├── jobs-and-webhooks2026-08-14
│ └── untrusted-content2026-08-14
├── endpoints/
│ ├── batch2026-08-15
│ ├── crawl2026-08-15
│ ├── logo2026-08-16
│ ├── scrape2026-08-15
│ └── search2026-08-14
├── help/
│ └── api-keys2026-08-16
├── introduction2026-08-14
└── quickstart2026-08-14
50 linksevery one with its last change1 credit
{"url": "docs.hydrafetch.com", "limit": 12}
docs.hydrafetch.com/
docs.hydrafetch.com/introduction
docs.hydrafetch.com/quickstart
docs.hydrafetch.com/changelog
docs.hydrafetch.com/concepts/credits
docs.hydrafetch.com/concepts/caching
docs.hydrafetch.com/concepts/errors
docs.hydrafetch.com/concepts/formats
docs.hydrafetch.com/concepts/jobs-and-webhooks
docs.hydrafetch.com/endpoints/scrape
docs.hydrafetch.com/endpoints/crawl
docs.hydrafetch.com/endpoints/search
12/12 pages0 failed1 credit a page
{"query": "retrieval augmented generation", "limit": 5, "scrapeResults": true}
aws.amazon.com1,512 words
What is RAG? - Retrieval-Augmented Generation AI Explained - AWS
3 days ago - What is Retrieval-Augmented Generation (RAG), how and why businesses use RAG AI, and how to use RAG with AWS.
ibm.com2,068 words
What is RAG (Retrieval Augmented Generation)? | IBM
October 31, 2024 - Retrieval augmented generation (RAG) is an architecture for optimizing the performance of an artificial intelligence (AI) model by connecting it with external knowledge bases.
reddit.com1,397 words
ELI5 What is a is Retrieval-Augmented Generation (RAG)
cloud.google.com924 words
What is Retrieval-Augmented Generation (RAG)? | Google Cloud
Retrieval-augmented generation (RAG) combines LLMs with external knowledge bases to improve their outputs. Learn more with Google Cloud.
blogs.nvidia.com1,630 words
What Is Retrieval-Augmented Generation aka RAG | NVIDIA Blogs
October 9, 2025 - Retrieval-augmented generation is a technique for enhancing the accuracy and reliability of generative AI models with information fetched from specific and relevant data sources.
5 resultsevery one fetched and cleaned6 credits
{"urls": ["en.wikipedia.org/wiki/List_of_largest_companies_by_revenue"], "schema": {…}}
#companyindustryrev $m
1AmazonRetail, Information technology716,000
2WalmartRetail713,000
3State Grid Corporation of ChinaElectricity545,000
4Saudi AramcoOil and gas480,000
5China National Petroleum CorporationOil and gas476,000
6China Petrochemical CorporationOil and gas429,000
7AppleInformation technology416,000
8AlphabetInformation technology402,000
9UnitedHealth GroupHealthcare400,000
10Berkshire HathawayFinancials371,000
11CVS HealthHealthcare357,000
12Volkswagen GroupAutomotive348,000
13ExxonMobilOil and gas344,000
14VitolCommodities331,000
15ShellOil and gas323,000
16China State Construction EngineeringConstruction320,000
17ToyotaAutomotive312,000
18McKessonHealthcare308,000
19MicrosoftInformation technology281,000
20CencoraHealthcare262,000
21TrafiguraCommodities244,000
22CostcoRetail242,000
23JPMorgan ChaseFinancials239,000
24Industrial and Commercial Bank of ChinaFinancials222,000
25Schwarz GruppeRetail220,000
26TotalEnergiesOil and gas218,000
27GlencoreCommodities217,000
28NvidiaSemiconductors215,000
29BPOil and gas213,000
30Cardinal HealthHealthcare205,000
50 records typed off one page5 credits
{"domain": "catchintent.com"}preview
CatchIntent
Book more meetings from in-market buyers
industryInformation · Software PublishersNAICS 513210
#000000
#2d2616
#667eea
#fee685
#f99c00
#00bb7f
fontsGeist · Geist Mono · Apple Color Emoji
socials[](https://www.linkedin.com/company/catchintent)[](https://x.com/catchintenthq)[](https://github.com/CatchIntent)[](https://www.instagram.com/catchintent)
4 assets6 colours ranked4 socials5 credits
img.hydrafetch.com/logo/catchintent.com?token=…preview
served from cachelocked to your domainsno credits
[02 / Behind one call]
## You call one endpoint. We handle the hard part.
No headless browsers to run, no proxies to bring, no retry logic to write. The response is clean or it costs you nothing.
JS-rendered pages
### Client-side apps come back as content.
React, Vue, anything that draws in the browser. You never see an empty shell where the page should be.
bundlephobia.com/package/react
what comes back
bundle size7.4 kB minified ·2.8 kB gzipped
download time57 ms slow 3G ·3 ms 4G
spinning up a browser
Blocks and challenges
### Challenge walls are our problem.
Pages that greet a bot with a wall come back as content anyway. When one truly will not yield, the response says so honestly.
403 · captcha
Verifying you are human…
Challenge cleared
what your agent sees
```
# software engineer jobs
Sort by: **relevance** - **date**
## Careers in DevOps engineering
### Senior DevOps Site Reliability Engineer
Reynolds and Reynolds
North Andover, MA 01845
- Referral program
- Parental leave
- 401(k) matching
- Health insurance
### DevOps Engineer
Triumph Tech, LLC
Surprise, AZ 85378
- 401(k)
- Health insurance
- Paid time off
- Vision insurance
### Senior Cloud Engineer
firstPRO
Hybrid work in Marlton, NJ 08053
```
"blocked": false · 2,090 words · 1 creditwhen one will not yield: 0 credits
Failures are free
Calls run, credits are charged for what came back, and a page we could not deliver never reaches your bill.
scrapecatchintent.com/pricing1 credit
brandcatchintent.com5 credits
scrapeg2.com/products/notion/reviewsnothing billed0 credits
mapdocs.hydrafetch.com1 credit
scrapebundlephobia.com/package/react1 credit
extracten.wikipedia.org/wiki/List_of_largest_companies5 credits
screenshotbundlephobia.com/package/react5 credits
scrapecaniuse.com/fetch1 credit
searchretrieval augmented generation6 credits
scrapecatchintent.com/pricingcached1 credit
scrapeg2.com/categories/web-scrapingnothing billed0 credits
scrapeindeed.com/q-software-engineer-jobs1 credit
scrapecatchintent.com/pricing1 credit
brandcatchintent.com5 credits
scrapeg2.com/products/notion/reviewsnothing billed0 credits
mapdocs.hydrafetch.com1 credit
scrapebundlephobia.com/package/react1 credit
extracten.wikipedia.org/wiki/List_of_largest_companies5 credits
screenshotbundlephobia.com/package/react5 credits
scrapecaniuse.com/fetch1 credit
searchretrieval augmented generation6 credits
scrapecatchintent.com/pricingcached1 credit
scrapeg2.com/categories/web-scrapingnothing billed0 credits
scrapeindeed.com/q-software-engineer-jobs1 credit
[03 / Integration]
## Call it from your stack.
An official client in 6 languages, or plain HTTP from anywhere.
[All SDKs](https://hydrafetch.com/integrations/#sdks)
```
curl https://api.hydrafetch.com/v1/web/scrape \
-H "X-API-Key: $HYDRAFETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://en.wikipedia.org/wiki/Web_scraping"}'
```
`$npm install @hydrafetch/node-sdk`
```
import { Hydrafetch } from "@hydrafetch/node-sdk";
const hf = new Hydrafetch(process.env.HYDRAFETCH_API_KEY);
const page = await hf.scrape("https://en.wikipedia.org/wiki/Web_scraping");
console.log(page.markdown);
```
`$pip install hydrafetch`
```
from hydrafetch import Hydrafetch
hf = Hydrafetch()
page = hf.scrape("https://en.wikipedia.org/wiki/Web_scraping")
print(page["markdown"])
```
`$go get github.com/Hydrafetch/go-sdk`
```
hf, err := hydrafetch.New("")
if err != nil {
log.Fatal(err)
}
page, err := hf.Scrape(context.Background(), "https://en.wikipedia.org/wiki/Web_scraping", nil)
if err != nil {
log.Fatal(err)
}
fmt.Println(page.Markdown)
```
`$gem install hydrafetch`
```
require "hydrafetch"
hf = Hydrafetch::Client.new
page = hf.scrape("https://en.wikipedia.org/wiki/Web_scraping")
puts page["markdown"]
```
`$cargo add hydrafetch`
```
use hydrafetch::Client;
#[tokio::main]
async fn main() -> Result<(), Box> {
let hf = Client::from_env()?;
let page = hf.scrape("https://en.wikipedia.org/wiki/Web_scraping", None).await?;
println!("{}", page.markdown.unwrap_or_default());
Ok(())
}
```
`$composer require hydrafetch/hydrafetch`
```
use Hydrafetch\Client;
$hf = new Client();
$page = $hf->scrape('https://en.wikipedia.org/wiki/Web_scraping');
echo $page['markdown'];
```
For pipelines
### Short calls return. Long ones hand you an id.
Requests wait inline by default, so most calls are a single round trip. A crawl or a batch hands back an id you poll instead. Wire either into a queue, a cron, or a nightly refresh and stop thinking about it.
[Read the docs](https://docs.hydrafetch.com/)
For agents
### Every endpoint is a tool.
Point your client at the MCP server and an agent can reach the live web mid-conversation to ground its own answers. Same primitives, same credits.
api.hydrafetch.com/mcpscrapemapsearchextract
Works withClaudeCursorCodex
[All integrations](https://hydrafetch.com/integrations/)
[04 / In practice]
## From raw web to product features.
6 ways teams wire it in. The envelope never changes.
the chain2 calls, one envelope
#callinreturnscost
1mapdocs.hydrafetch.com50 urls, each with its last change1
2batchthe 12 that changed12 pages of clean markdown12
```
**Published:** 2005-09-17
# Web scraping
**Web scraping**, **web harvesting**, or **web data extraction** is data scraping used for extracting data from websites.
Scraping a web page involves fetching it and then extracting data from it. Fetching is the downloading of a page.
Web pages are built using text-based markup languages (HTML and XHTML), and frequently contain useful data in text form.
## History
After the birth of the World Wide Web in 1989, the first web robot was created in June 1993.
In December 1993, the first crawler-based web search engine, JumpStation, was launched.
## Techniques
Data extraction techniques range from manual collection to sophisticated automation.
### Human copy-and-paste
```
Chunk, embed, upsert. Tomorrow only the changed pages cost anything.
13 credits in totalsame response shape every call
the chain2 calls, one envelope
#callinreturnscost
1search"retrieval augmented generation"5 ranked results1
2scrapeall 5, in the same call5 pages of markdown, cited5
What is RAG? - Retrieval-Augmented Generation AI Explained
aws.amazon.com · 1,512 words
What is RAG (Retrieval Augmented Generation)? | IBM
ibm.com · 2,068 words
ELI5 What is a is Retrieval-Augmented Generation (RAG)
reddit.com · 1,397 words
What is Retrieval-Augmented Generation (RAG)? | Google Cloud
cloud.google.com · 924 words
What Is Retrieval-Augmented Generation aka RAG | NVIDIA Blogs
blogs.nvidia.com · 1,630 words
One round trip. The agent answers from pages that exist today.
6 credits in totalsame response shape every call
the chain1 call, one envelope
#callinreturnscost
1extractone listing page + a schema50 typed records, no nulls guessed5
1AmazonRetail716,000
2WalmartRetail713,000
3State Grid Corporation of ChinaElectricity545,000
4Saudi AramcoOil and gas480,000
5China National Petroleum CorporationOil and gas476,000
6China Petrochemical CorporationOil and gas429,000
7AppleInformation technology416,000
8AlphabetInformation technology402,000
9UnitedHealth GroupHealthcare400,000
10Berkshire HathawayFinancials371,000
11CVS HealthHealthcare357,000
12Volkswagen GroupAutomotive348,000
A directory becomes rows. Fields it cannot find come back null, never invented.
5 credits in totalsame response shape every call
the chain1 call, one envelope
#callinreturnscost
1brandcatchintent.com4 assets, 6 colours, 4 socials, NAICS5
CatchIntent
Information · Software Publishers · NAICS 513210
#000000
#2d2616
#667eea
#fee685
#f99c00
#00bb7f
fontsGeist · Geist Mono
socials
taglineBook more meetings from in-market buyers
Enough to render the company in your UI without a designer touching it.
5 credits in totalsame response shape every call
the chain1 call, one envelope
#callinreturnscost
1batch20 product urls, every morning20 pages, 20 quality scores20
catchintent.com/pricing656 wordschanged
catchintent.com/dfyfrom cachesame
catchintent.com/demofrom cachesame
catchintent.com/blog2 new postschanged
catchintent.com/docsfrom cachesame
catchintent.com/terms-of-servicefrom cachesame
catchintent.com/privacy-policyfrom cachesame
catchintent.com/contactfrom cachesame
Same envelope every run, so a diff is a diff and not a parser change.
20 credits in totalsame response shape every call
the chain1 call, one envelope
#callinreturnscost
1mapdocs.astro.build242 urls, each with its last change1
docs.astro.build/en/getting-started2026-08-21
docs.astro.build/en/concepts/islands2026-08-19
docs.astro.build/en/guides/content-collections2026-08-18
docs.astro.build/en/reference/configuration2026-08-14
docs.astro.build/en/guides/deploy2026-08-11
docs.astro.build/en/guides/integrations-guide2026-08-04
One credit for the whole inventory. Then crawl only what actually moved.
1 credit in totalsame response shape every call
[05 / Second opinions]
## Do not take our word for it. Ask an AI to check.
Point your coding agent at your own repository, or ask an assistant to compare us against the alternatives. Both answer with things we did not write.
Ask your coding agent
### It reads your repo, not our pitch.
Paste one line into Claude Code, Cursor, or anything that reads your repository. It comes back with file paths, a credit estimate, and where we are not worth it.
The skill reads our live endpoint list when it runs, so it covers anything we ship after you paste this. [Read it first](https://hydrafetch.com/.well-known/agent-skills/where-to-use-hydrafetch/SKILL.md).
Ask an assistant
### Let it compare us to the alternatives.
The question is already written, and it asks for the comparison rather than the sales pitch. It opens in a new tab with the prompt filled in.
[Ask ChatGPT](https://chatgpt.com/?q=I)[Ask Claude](https://claude.ai/new?q=I)[Ask Gemini](https://www.google.com/search?udm=50&aep=11&q=I)[Ask Perplexity](https://www.perplexity.ai/search/new?q=I)
[Talk to a human](https://hydrafetch.com/demo/)Assistants read the live site, so they answer from what is published today rather than from anything we hand them.
[06 / Questions]
## Asked before you ask.
The short answers. The long ones live in the [docs](https://docs.hydrafetch.com/).
Getting started
Sign up at the dashboard and your key is on the first screen, with 250 free credits attached. No card required.
Yes, because keys and credits belong to a workspace. Sign up with an email address and password, or with Google. Signing in with Google is optional and uses your name and email address only, to create and secure your Hydrafetch account. We never read, send, or store anything else from your Google account.
Yes, for Node, Python, Go, Ruby, Rust and PHP, plus two browser packages for company logos. Each one wraps the same HTTP surface, so plain HTTP still works from any language in a few lines, and agents can skip HTTP entirely and connect over MCP.
Capabilities
Yes. Pages that only exist after the browser draws them come back as clean content, automatically, for the same one credit.
No. Blocks, challenges and rate limits are our problem. If a page truly cannot be delivered, the response says so and you are not billed.
Ask for it. Every scrape can return the page’s own structured data alongside Markdown, and the extract endpoint turns any page into typed JSON matching your schema, with per-field evidence.
Yes. Crawls run as asynchronous jobs with depth, path and subdomain controls, and a webhook tells you when they finish. Every page arrives in the same envelope.
Fresh enough for the job. Recently seen pages are served from a short-lived cache and refreshed when they go stale. If your use case has a stricter bar, tell the request how fresh is fresh enough and we take care of the rest. Every response states whether it came from cache.
Billing
A successful page costs one credit, whatever it took to get it. Search costs one plus one per scraped result, and extract costs five. There are no modifiers or surcharges to stack.
Never. Errors, blocks and dead ends cost nothing. You pay for delivered pages only.
No. What a page needed to come back clean is our cost, not your line item. One page, one credit.
## Clean web data is one call away.
250 free credits, no card required. Failures never billed.
[Get API key](https://app.hydrafetch.com/)[Read the docs](https://docs.hydrafetch.com/)
[ Pages processed ]
Domains reached
Searches run
Sites crawled