tinker_cookbook.tokenizer_utils.get_tokenizer

tinker_cookbook.tokenizer_utils.get_tokenizer(model_name)

Get a tokenizer by name.

Parameters:

• model_name (str) – HuggingFace model identifier (e.g. "Qwen/Qwen3-8B") or a custom registered name.

Returns: Tokenizer: A PreTrainedTokenizer instance.