tinker_cookbook.tokenizer_utils

Class	Description
TmlTokenizer	Structural type for the underlying tml_renderers tokenizer.
SupportsTmlTokenizer	Tokenizer facade that exposes an underlying TmlTokenizer.
TmlRenderersTokenizerAdapter	Small tokenizer facade for tml_renderers models used through cookbook.

Function	Description
ensure_tml_renderers_importable()	Make tml_renderers importable, falling back to TML_RENDERERS_SOURCE_DIR.
register_tokenizer()	Register a custom tokenizer factory.
get_registered_tokenizer_names()	Return a list of all registered custom tokenizer names.
is_tokenizer_registered()	Check if a tokenizer name is registered.
unregister_tokenizer()	Unregister a custom tokenizer factory.
get_tokenizer()	Get a tokenizer by name.