UnstructuredLakeFSLoader#
- class langchain_community.document_loaders.lakefs.UnstructuredLakeFSLoader(url: str, repo: str, ref: str = 'main', path: str = '', presign: bool = True, **unstructured_kwargs: Any)[source]#
Load from lakeFS as unstructured data.
Initialize UnstructuredLakeFSLoader.
Args:
- Parameters:
lakefs_access_key β
lakefs_secret_key β
lakefs_endpoint β
repo (str) β
ref (str) β
url (str) β
path (str) β
presign (bool) β
unstructured_kwargs (Any) β
Methods
__init__
(url,Β repo[,Β ref,Β path,Β presign])Initialize UnstructuredLakeFSLoader.
A lazy loader for Documents.
aload
()Load data into Document objects.
Load file.
load
()Load data into Document objects.
load_and_split
([text_splitter])Load Documents and split into chunks.
- __init__(url: str, repo: str, ref: str = 'main', path: str = '', presign: bool = True, **unstructured_kwargs: Any)[source]#
Initialize UnstructuredLakeFSLoader.
Args:
- Parameters:
lakefs_access_key β
lakefs_secret_key β
lakefs_endpoint β
repo (str) β
ref (str) β
url (str) β
path (str) β
presign (bool) β
unstructured_kwargs (Any) β
- async alazy_load() AsyncIterator[Document] #
A lazy loader for Documents.
- Return type:
AsyncIterator[Document]
- load_and_split(text_splitter: TextSplitter | None = None) List[Document] #
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
- Parameters:
text_splitter (Optional[TextSplitter]) β TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.
- Returns:
List of Documents.
- Return type:
List[Document]