ScrapingAntLoader#

class langchain_community.document_loaders.scrapingant.ScrapingAntLoader(urls: List[str], *, api_key: str | None = None, scrape_config: dict | None = None, continue_on_failure: bool = True)[source]#

Turn an url to LLM accessible markdown with ScrapingAnt.

For further details, visit: https://docs.scrapingant.com/python-client

Initialize client.

Parameters:

  • urls (List[str]) – List of urls to scrape.

  • api_key (str | None) – The ScrapingAnt API key. If not specified must have env var SCRAPINGANT_API_KEY set.

  • scrape_config (dict | None) – The scraping config from ScrapingAntClient.markdown_request

  • continue_on_failure (bool) – Whether to continue if scraping an url fails.

Methods

__init__(urls, *[, api_key, scrape_config, ...])

Initialize client.

alazy_load()

A lazy loader for Documents.

aload()

Load data into Document objects.

lazy_load()

Fetch data from ScrapingAnt.

load()

Load data into Document objects.

load_and_split([text_splitter])

Load Documents and split into chunks.
__init__(urls: List[str], *, api_key: str | None = None, scrape_config: dict | None = None, continue_on_failure: bool = True) → None[source]#

Initialize client.

Parameters:

  • urls (List[str]) – List of urls to scrape.

  • api_key (str | None) – The ScrapingAnt API key. If not specified must have env var SCRAPINGANT_API_KEY set.

  • scrape_config (dict | None) – The scraping config from ScrapingAntClient.markdown_request

  • continue_on_failure (bool) – Whether to continue if scraping an url fails.

Return type:

None

async alazy_load() → AsyncIterator[Document]#

A lazy loader for Documents.

Return type:

AsyncIterator[Document]

async aload() → List[Document]#

Load data into Document objects.

Return type:

List[Document]

lazy_load() → Iterator[Document][source]#

Fetch data from ScrapingAnt.

Return type:

Iterator[Document]

load() → List[Document]#

Load data into Document objects.

Return type:

List[Document]

load_and_split(text_splitter: TextSplitter | None = None) → List[Document]#

Load Documents and split into chunks. Chunks are returned as Documents.

Do not override this method. It should be considered to be deprecated!

Parameters:

text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns:

List of Documents.

Return type:

List[Document]

Examples using ScrapingAntLoader

On this page