`base`#

Classes

`base.Language`(value)	Enum of the programming languages.
`base.TextSplitter`(chunk_size, chunk_overlap, ...)	Interface for splitting text into chunks.
`base.TokenTextSplitter`([encoding_name, ...])	Splitting text to tokens using model tokenizer.
`base.Tokenizer`(chunk_overlap, ...)	Tokenizer data class.

Functions

base.split_text_on_tokens(*, text, tokenizer)

Split incoming text and return chunks using tokenizer.

base#