Copy as Markdown View as Markdown

tinker_cookbook.tokenizer_utils.encode_without_special_tokens

Generated from tinker-cookbook at commit 8486b8d. Source links point at that snapshot.

tinker_cookbook.tokenizer_utils.encode_without_special_tokens(tokenizer, text)

Encode raw text without adding model-level BOS/EOS tokens.

Parameters:

• tokenizer (Tokenizer | SupportsEncodeOrdinary)
• text (str)

Returns: list[int]