Skip to content
View as Markdown

tinker_cookbook.hyperparam_utils.transfer_lr

Generated from tinker-cookbook at commit 28de41e. Source links point at that snapshot.

tinker_cookbook.hyperparam_utils.transfer_lr(lr, from_model, to_model)

Translate an AdamW LoRA learning rate tuned on one model to another model.

The result is lr * mult(to_model) / mult(from_model). It assumes everything else (data, batch size, steps, LoRA rank, optimizer) stays the same. This is the main use of the multipliers: the learning rate you tuned carries your task's absolute scale, which get_lr can only guess. These model ratios are calibrated for AdamW; for Dimuon, tune on the target model directly. Optimizer conversion is a separate estimate with its own scope in the hyperparameter guide.

Parameters:

  • lr (float) – Learning rate known to work well for from_model.
  • from_model (str) – Model the learning rate was tuned on.
  • to_model (str) – Model to translate the learning rate to.

Returns: float – Learning rate for to_model.

Referenced by