tinker_cookbook.hyperparam_utils.transfer_lr
Generated from tinker-cookbook at commit 28de41e. Source links point at that snapshot.
tinker_cookbook.hyperparam_utils.transfer_lr(lr, from_model, to_model)
Translate an AdamW LoRA learning rate tuned on one model to another model.
The result is lr * mult(to_model) / mult(from_model). It assumes everything
else (data, batch size, steps, LoRA rank, optimizer) stays the same. This is the
main use of the multipliers: the learning rate you tuned carries your task's
absolute scale, which get_lr can only guess. These model ratios are
calibrated for AdamW; for Dimuon, tune on the target model directly. Optimizer
conversion is a separate estimate with its own scope in the hyperparameter guide.
Parameters:
- lr (float) – Learning rate known to work well for
from_model. - from_model (str) – Model the learning rate was tuned on.
- to_model (str) – Model to translate the learning rate to.
Returns: float – Learning rate for to_model.