UnstructuredLoader#

class langchain_unstructured.document_loaders.UnstructuredLoader(file_path: str | Path | list[str] | list[Path] | None = None, *, file: IO[bytes] | list[IO[bytes]] | None = None, partition_via_api: bool = False, post_processors: list[Callable[[str], str]] | None = None, api_key: str | None = None, client: UnstructuredClient | None = None, url: str | None = None, web_url: str | None = None, **kwargs: Any)[source]#

Unstructured document loader interface.

Setup:

Install langchain-unstructured and set environment variable UNSTRUCTURED_API_KEY.

Instantiate:
Lazy load:
docs = []
docs_lazy = loader.lazy_load()

# async variant:
# docs_lazy = await loader.alazy_load()

for doc in docs_lazy:
    docs.append(doc)
print(docs[0].page_content[:100])
print(docs[0].metadata)

1 2 0 2
{'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-07-25T21:28:58', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}
Async load:
docs = await loader.aload()
print(docs[0].page_content[:100])
print(docs[0].metadata)

1 2 0 2
{'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-07-25T21:28:58', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}
Load URL:
loader = UnstructuredLoader(web_url="https://www.example.com/")
print(docs[0])

page_content='Example Domain' metadata={'category_depth': 0, 'languages': ['eng'], 'filetype': 'text/html', 'url': 'https://www.example.com/', 'category': 'Title', 'element_id': 'fdaa78d856f9d143aeeed85bf23f58f8'}

print(docs[1])

page_content='This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.' metadata={'languages': ['eng'], 'parent_id': 'fdaa78d856f9d143aeeed85bf23f58f8', 'filetype': 'text/html', 'url': 'https://www.example.com/', 'category': 'NarrativeText', 'element_id': '3652b8458b0688639f973fe36253c992'}

References

https://docs.unstructured.io/api-reference/api-services/sdk https://docs.unstructured.io/api-reference/api-services/overview https://docs.unstructured.io/open-source/core-functionality/partitioning https://docs.unstructured.io/open-source/core-functionality/chunking

Initialize loader.

Methods

__init__([file_path, file, ...])

Initialize loader.

alazy_load()

A lazy loader for Documents.

aload()

Load data into Document objects.

lazy_load()

Load file(s) to the _UnstructuredBaseLoader.

load()

Load data into Document objects.

load_and_split([text_splitter])

Load Documents and split into chunks.
Parameters:

  • file_path (Optional[str | Path | list[str] | list[Path]])

  • file (Optional[IO[bytes] | list[IO[bytes]]])

  • partition_via_api (bool)

  • post_processors (Optional[list[Callable[[str], str]]])

  • api_key (Optional[str])

  • client (Optional[UnstructuredClient])

  • url (Optional[str])

  • web_url (Optional[str])

  • kwargs (Any)

__init__(file_path: str | Path | list[str] | list[Path] | None = None, *, file: IO[bytes] | list[IO[bytes]] | None = None, partition_via_api: bool = False, post_processors: list[Callable[[str], str]] | None = None, api_key: str | None = None, client: UnstructuredClient | None = None, url: str | None = None, web_url: str | None = None, **kwargs: Any)[source]#

Initialize loader.

Parameters:

  • file_path (str | Path | list[str] | list[Path] | None)

  • file (IO[bytes] | list[IO[bytes]] | None)

  • partition_via_api (bool)

  • post_processors (list[Callable[[str], str]] | None)

  • api_key (str | None)

  • client (UnstructuredClient | None)

  • url (str | None)

  • web_url (str | None)

  • kwargs (Any)

async alazy_load() → AsyncIterator[Document]#

A lazy loader for Documents.

Return type:

AsyncIterator[Document]

async aload() → list[Document]#

Load data into Document objects.

Return type:

list[Document]

lazy_load() → Iterator[Document][source]#

Load file(s) to the _UnstructuredBaseLoader.

Return type:

Iterator[Document]

load() → list[Document]#

Load data into Document objects.

Return type:

list[Document]

load_and_split(text_splitter: TextSplitter | None = None) → list[Document]#

Load Documents and split into chunks. Chunks are returned as Documents.

Do not override this method. It should be considered to be deprecated!

Parameters:

text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns:

List of Documents.

Return type:

list[Document]