PythonCodeTextSplitter#
- class langchain_text_splitters.python.PythonCodeTextSplitter(**kwargs: Any)[source]#
Attempts to split the text along Python syntax.
Initialize a PythonCodeTextSplitter.
Methods
__init__(**kwargs)Initialize a PythonCodeTextSplitter.
atransform_documents(documents, **kwargs)Asynchronously transform a list of documents.
create_documents(texts[, metadatas])Create documents from a list of texts.
from_huggingface_tokenizer(tokenizer, **kwargs)Text splitter that uses HuggingFace tokenizer to count length.
from_language(language, **kwargs)Return an instance of this class based on a specific language.
from_tiktoken_encoder([encoding_name, ...])Text splitter that uses tiktoken encoder to count length.
get_separators_for_language(language)Retrieve a list of separators specific to the given language.
split_documents(documents)Split documents.
split_text(text)Split the input text into smaller chunks based on predefined separators.
transform_documents(documents, **kwargs)Transform sequence of documents by splitting them.
- Parameters:
kwargs (Any)
- __init__(**kwargs: Any) None[source]#
Initialize a PythonCodeTextSplitter.
- Parameters:
kwargs (Any)
- Return type:
None
- async atransform_documents(documents: Sequence[Document], **kwargs: Any) Sequence[Document]#
Asynchronously transform a list of documents.
- create_documents(texts: List[str], metadatas: List[dict] | None = None) List[Document]#
Create documents from a list of texts.
- Parameters:
texts (List[str])
metadatas (List[dict] | None)
- Return type:
List[Document]
- classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) TextSplitter#
Text splitter that uses HuggingFace tokenizer to count length.
- Parameters:
tokenizer (Any)
kwargs (Any)
- Return type:
- classmethod from_language(language: Language, **kwargs: Any) RecursiveCharacterTextSplitter#
Return an instance of this class based on a specific language.
This method initializes the text splitter with language-specific separators.
- Parameters:
language (Language) – The language to configure the text splitter for.
**kwargs (Any) – Additional keyword arguments to customize the splitter.
- Returns:
An instance of the text splitter configured for the specified language.
- Return type:
- classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: str | None = None, allowed_special: Literal['all'] | AbstractSet[str] = {}, disallowed_special: Literal['all'] | Collection[str] = 'all', **kwargs: Any) TS#
Text splitter that uses tiktoken encoder to count length.
- Parameters:
encoding_name (str)
model_name (str | None)
allowed_special (Literal['all'] | ~typing.AbstractSet[str])
disallowed_special (Literal['all'] | ~typing.Collection[str])
kwargs (Any)
- Return type:
TS
- static get_separators_for_language(language: Language) List[str]#
Retrieve a list of separators specific to the given language.
- Parameters:
language (Language) – The language for which to get the separators.
- Returns:
A list of separators appropriate for the specified language.
- Return type:
List[str]
- split_text(text: str) List[str]#
Split the input text into smaller chunks based on predefined separators.
- Parameters:
text (str) – The input text to be split.
- Returns:
A list of text chunks obtained after splitting.
- Return type:
List[str]