DocusaurusLoader#
- class langchain_community.document_loaders.docusaurus.DocusaurusLoader(url: str, custom_html_tags: List[str] | None = None, **kwargs: Any)[source]#
Load from Docusaurus Documentation.
It leverages the SitemapLoader to loop through the generated pages of a Docusaurus Documentation website and extracts the content by looking for specific HTML tags. By default, the parser searches for the main content of the Docusaurus page, which is normally the <article>. You can also define your own custom HTML tags by providing them as a list, for example: [βdivβ, β.mainβ, βaβ].
Initialize DocusaurusLoader
- Parameters:
url (str) β The base URL of the Docusaurus website.
custom_html_tags (List[str] | None) β Optional custom html tags to extract content from pages.
kwargs (Any) β Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure
Attributes
web_path
Methods
__init__
(url[,Β custom_html_tags])Initialize DocusaurusLoader
A lazy loader for Documents.
aload
()Load text from the urls in web_path async into Documents.
fetch_all
(urls)Fetch all urls concurrently with rate limiting.
Load sitemap.
load
()Load data into Document objects.
load_and_split
([text_splitter])Load Documents and split into chunks.
parse_sitemap
(soup,Β *[,Β depth])Parse sitemap xml and load into a list of dicts.
scrape
([parser])Scrape data from webpage and return it in BeautifulSoup format.
scrape_all
(urls[,Β parser])Fetch all urls, then return soups for all results.
- __init__(url: str, custom_html_tags: List[str] | None = None, **kwargs: Any)[source]#
Initialize DocusaurusLoader
- Parameters:
url (str) β The base URL of the Docusaurus website.
custom_html_tags (List[str] | None) β Optional custom html tags to extract content from pages.
kwargs (Any) β Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure
- async alazy_load() AsyncIterator[Document] #
A lazy loader for Documents.
- Return type:
AsyncIterator[Document]
- aload() List[Document] #
Load text from the urls in web_path async into Documents.
- Return type:
List[Document]
- async fetch_all(urls: List[str]) Any #
Fetch all urls concurrently with rate limiting.
- Parameters:
urls (List[str]) β
- Return type:
Any
- load_and_split(text_splitter: TextSplitter | None = None) List[Document] #
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
- Parameters:
text_splitter (Optional[TextSplitter]) β TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.
- Returns:
List of Documents.
- Return type:
List[Document]
- parse_sitemap(soup: Any, *, depth: int = 0) List[dict] #
Parse sitemap xml and load into a list of dicts.
- Parameters:
soup (Any) β BeautifulSoup object.
depth (int) β current depth of the sitemap. Default: 0
- Returns:
List of dicts.
- Return type:
List[dict]
- scrape(parser: str | None = None) Any #
Scrape data from webpage and return it in BeautifulSoup format.
- Parameters:
parser (str | None) β
- Return type:
Any
- scrape_all(urls: List[str], parser: str | None = None) List[Any] #
Fetch all urls, then return soups for all results.
- Parameters:
urls (List[str]) β
parser (str | None) β
- Return type:
List[Any]
Examples using DocusaurusLoader