ScrapflyLoader#

class langchain_community.document_loaders.scrapfly.ScrapflyLoader(urls: List[str], *, api_key: str | None = None, scrape_format: Literal['markdown', 'text'] = 'markdown', scrape_config: dict | None = None, continue_on_failure: bool = True)[source]#

Turn a url to llm accessible markdown with Scrapfly.io.

For further details, visit: https://scrapfly.io/docs/sdk/python

Initialize client.

Parameters:

  • urls (List[str]) – List of urls to scrape.

  • api_key (str | None) – The Scrapfly API key. If not specified must have env var SCRAPFLY_API_KEY set.

  • scrape_format (Literal['markdown', 'text']) – Scrape result format, one or “markdown” or “text”.

  • scrape_config (dict | None) – Dictionary of ScrapFly scrape config object.

  • continue_on_failure (bool) – Whether to continue if scraping a url fails.

Methods

__init__(urls, *[, api_key, scrape_format, ...])

Initialize client.

alazy_load()

A lazy loader for Documents.

aload()

Load data into Document objects.

lazy_load()

A lazy loader for Documents.

load()

Load data into Document objects.

load_and_split([text_splitter])

Load Documents and split into chunks.
__init__(urls: List[str], *, api_key: str | None = None, scrape_format: Literal['markdown', 'text'] = 'markdown', scrape_config: dict | None = None, continue_on_failure: bool = True) → None[source]#

Initialize client.

Parameters:

  • urls (List[str]) – List of urls to scrape.

  • api_key (str | None) – The Scrapfly API key. If not specified must have env var SCRAPFLY_API_KEY set.

  • scrape_format (Literal['markdown', 'text']) – Scrape result format, one or “markdown” or “text”.

  • scrape_config (dict | None) – Dictionary of ScrapFly scrape config object.

  • continue_on_failure (bool) – Whether to continue if scraping a url fails.

Return type:

None

async alazy_load() → AsyncIterator[Document]#

A lazy loader for Documents.

Return type:

AsyncIterator[Document]

async aload() → list[Document]#

Load data into Document objects.

Return type:

list[Document]

lazy_load() → Iterator[Document][source]#

A lazy loader for Documents.

Return type:

Iterator[Document]

load() → list[Document]#

Load data into Document objects.

Return type:

list[Document]

load_and_split(text_splitter: TextSplitter | None = None) → list[Document]#

Load Documents and split into chunks. Chunks are returned as Documents.

Do not override this method. It should be considered to be deprecated!

Parameters:

text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns:

List of Documents.

Return type:

list[Document]

Examples using ScrapflyLoader