LangChain + Hydrafetch

Web pages as LangChain documents.

A document loader that hands your splitter clean markdown, a retriever that searches the web, and tools an agent can call. One page, a whole site or a search, in the shapes LangChain already uses.

LangChain

with Hydrafetch

installpip install langchain-hydrafetch
loader modesscrape · crawl · map
retrieverweb search
agent tools6
sync and asyncboth

[ 01 / Set up ]

Install, set a key, load a page.

The package reads your key from the environment, so nothing else needs configuring.

Load pages

  1. 1

    Install the package.

    Terminal
    pip install langchain-hydrafetch
  2. 2

    Set your API key from the dashboard.

    Terminal
    export HYDRAFETCH_API_KEY="hf_your_key"
  3. 3

    Load one page, or a whole site.

    loader.py
    from langchain_hydrafetch import HydrafetchLoader
    
    docs = HydrafetchLoader("https://example.com/article").load()
    
    site = HydrafetchLoader(
        "https://example.com",
        mode="crawl",
        params={"limit": 50},
    ).load()

Search and agent tools

  1. 1

    Search the web as a retriever. Ask for the full page text when a snippet is not enough.

    retriever.py
    from langchain_hydrafetch import HydrafetchSearchRetriever
    
    retriever = HydrafetchSearchRetriever(k=5, scrape_content=True)
    docs = retriever.invoke("best open source vector databases")
  2. 2

    Give an agent tools to search, read, extract and look up companies, jobs and news.

    tools.py
    from langchain_hydrafetch import (
        HydrafetchScrapeTool,
        HydrafetchSearchTool,
    )
    
    tools = [HydrafetchSearchTool(), HydrafetchScrapeTool()]

Search finds pages and scrape reads one you already have. Giving an agent both is the usual setup.

[ 02 / What to ask ]

Three pieces, in LangChain's own shapes.

Each one maps to an API call, and costs what that call costs.

  • HydrafetchLoader in scrape mode: one page as a document.

    loader1 credit
  • HydrafetchLoader in crawl mode: one document per page of a site.

    loader1 credit per page
  • HydrafetchLoader in map mode: one document per address found, without reading the pages.

    loader1 credit
  • HydrafetchSearchRetriever: search results as documents.

    retriever1 credit
  • HydrafetchExtractTool: fields from pages as JSON, from a schema or a description.

    tool5 credits per URL

[ 03 / What comes back ]

What a loaded document carries.

A real run of the loader against one documentation page. The page content came back as 1,519 words of markdown, with this metadata on the document.

HydrafetchLoader(url).load()[0].metadata1,519 words in page_content
sourcehttps://nextjs.org/docs/app/getting-started/installation
status200
titleInstallation
languageen
site_namenextjs.org
modified_time2026-10-02
word_count1519
page_typedocumentation

run with langchain-hydrafetch on 2026-11-17

[ 04 / Questions ]

What people ask about LangChain and Hydrafetch.

Install langchain-hydrafetch and call HydrafetchLoader with the page address. It returns a document whose page_content is the page as clean markdown and whose metadata carries the source, title, language and page type.

Yes. Pass mode="crawl" to the loader and it returns one document per page found. Use lazy_load to stream a large crawl instead of holding every page in memory.

Yes. HydrafetchSearchRetriever returns search results as documents. Pass scrape_content=True to get the full page text for each result instead of the snippet.

The package is free and open source. Calls cost the same credits as the API: 1 credit for a page, 1 credit per page crawled, and 1 credit for a search plus 1 credit for each result read in full.

The loader raises instead of returning a document, so the body of an error page never reaches your index. In crawl mode error pages are dropped and the rest of the crawl is kept.

[ 05 / Keep going ]

Other ways to connect.

[ Start ]

Clean web data is one call away.

500 free credits, no card required. Failures are never billed.

Success rate

 

Median scrape

 ms