Tokenizer#
- class langchain_text_splitters.base.Tokenizer(chunk_overlap: int, tokens_per_chunk: int, decode: Callable[[List[int]], str], encode: Callable[[str], List[int]])[source]#
Tokenizer data class.
Attributes
Methods
__init__
(chunk_overlap,Β tokens_per_chunk,Β ...)- Parameters:
chunk_overlap (int) β
tokens_per_chunk (int) β
decode (Callable[[List[int]], str]) β
encode (Callable[[str], List[int]]) β
- __init__(chunk_overlap: int, tokens_per_chunk: int, decode: Callable[[List[int]], str], encode: Callable[[str], List[int]]) None #
- Parameters:
chunk_overlap (int) β
tokens_per_chunk (int) β
decode (Callable[[List[int]], str]) β
encode (Callable[[str], List[int]]) β
- Return type:
None