tinker_cookbook.tokenizer_utils.register_tokenizer

tinker_cookbook.tokenizer_utils.register_tokenizer(name, factory)

Register a custom tokenizer factory.

Parameters:

• name (str) – The tokenizer name (typically a HuggingFace model ID like "Foo/foo_tokenizer").
• factory (Callable[[], Tokenizer]) – A callable that takes no arguments and returns a Tokenizer instance.