tinker_cookbook.tokenizer_utils
Generated from tinker-cookbook at commit 8486b8d. Source links point at that snapshot.
| Class | Description |
|---|---|
TmlTokenizer |
Structural type for the underlying tml_renderers tokenizer. |
SupportsEncodeOrdinary |
|
SupportsTmlTokenizer |
Tokenizer facade that exposes an underlying TmlTokenizer. |
TmlRenderersTokenizerAdapter |
Small tokenizer facade for tml_renderers models used through cookbook. |
| Function | Description |
|---|---|
ensure_tml_renderers_importable() |
Make tml_renderers importable, falling back to TML_RENDERERS_SOURCE_DIR. |
encode_without_special_tokens() |
Encode raw text without adding model-level BOS/EOS tokens. |
is_tml_renderers_model() |
Return whether tml-renderers tokenizes model_name. |
register_tokenizer() |
Register a custom tokenizer factory. |
get_registered_tokenizer_names() |
Return a list of all registered custom tokenizer names. |
is_tokenizer_registered() |
Check if a tokenizer name is registered. |
unregister_tokenizer() |
Unregister a custom tokenizer factory. |
get_tokenizer() |
Get a tokenizer by name. |