LangChain + Hydrafetch
Web pages as LangChain documents.
A document loader that hands your splitter clean markdown, a retriever that searches the web, and tools an agent can call. One page, a whole site or a search, in the shapes LangChain already uses.

LangChain
with Hydrafetch
[ 01 / Set up ]
Install, set a key, load a page.
The package reads your key from the environment, so nothing else needs configuring.
Load pages
- 1
Install the package.
Terminalpip install langchain-hydrafetch - 2
Set your API key from the dashboard.
Terminalexport HYDRAFETCH_API_KEY="hf_your_key" - 3
Load one page, or a whole site.
loader.pyfrom langchain_hydrafetch import HydrafetchLoader docs = HydrafetchLoader("https://example.com/article").load() site = HydrafetchLoader( "https://example.com", mode="crawl", params={"limit": 50}, ).load()
Search and agent tools
- 1
Search the web as a retriever. Ask for the full page text when a snippet is not enough.
retriever.pyfrom langchain_hydrafetch import HydrafetchSearchRetriever retriever = HydrafetchSearchRetriever(k=5, scrape_content=True) docs = retriever.invoke("best open source vector databases") - 2
Give an agent tools to search, read, extract and look up companies, jobs and news.
tools.pyfrom langchain_hydrafetch import ( HydrafetchScrapeTool, HydrafetchSearchTool, ) tools = [HydrafetchSearchTool(), HydrafetchScrapeTool()]
Search finds pages and scrape reads one you already have. Giving an agent both is the usual setup.
[ 02 / What to ask ]
Three pieces, in LangChain's own shapes.
Each one maps to an API call, and costs what that call costs.
HydrafetchLoader in scrape mode: one page as a document.
loader1 creditHydrafetchLoader in crawl mode: one document per page of a site.
loader1 credit per pageHydrafetchLoader in map mode: one document per address found, without reading the pages.
loader1 creditHydrafetchSearchRetriever: search results as documents.
retriever1 creditHydrafetchExtractTool: fields from pages as JSON, from a schema or a description.
tool5 credits per URL
[ 03 / What comes back ]
What a loaded document carries.
A real run of the loader against one documentation page. The page content came back as 1,519 words of markdown, with this metadata on the document.
run with langchain-hydrafetch on 2026-11-17
[ 04 / Questions ]
What people ask about LangChain and Hydrafetch.
Install langchain-hydrafetch and call HydrafetchLoader with the page address. It returns a document whose page_content is the page as clean markdown and whose metadata carries the source, title, language and page type.
Yes. Pass mode="crawl" to the loader and it returns one document per page found. Use lazy_load to stream a large crawl instead of holding every page in memory.
Yes. HydrafetchSearchRetriever returns search results as documents. Pass scrape_content=True to get the full page text for each result instead of the snippet.
The package is free and open source. Calls cost the same credits as the API: 1 credit for a page, 1 credit per page crawled, and 1 credit for a search plus 1 credit for each result read in full.
The loader raises instead of returning a document, so the body of an error page never reaches your index. In crawl mode error pages are dropped and the rest of the crawl is kept.
[ 05 / Keep going ]
Other ways to connect.
[ Start ]
Clean web data is one call away.
500 free credits, no card required. Failures are never billed.
Success rate
Median scrape
ms