Inflight Sampling
By default, save_weights_and_get_sampling_client returns a snapshot sampling client, which always samples from the weights saved when it was created. In an async RL loop that publishes new weights every step, rollouts that are already running keep sampling from the weights they started with until they finish, so long rollouts fall further and further behind the policy being trained.
Passing kind="inflight" instead makes every inflight sampling client for a training run share one set of weights that later inflight saves update in place. Running rollouts move to the newer weights as they are published, without being restarted and without losing their KV cache.
Use inflight sampling for rollouts during training, especially long multi-turn or agentic rollouts where staleness hurts. Keep the default snapshot kind wherever inference must stay on one set of weights, such as evaluations, checkpoint comparisons, and reproducible sampling.
Usage
# Publish W1 and use sampler_a for the rollouts starting now.
sampler_a = training_client.save_weights_and_get_sampling_client(kind="inflight")
# ... train further ...
# Publish W2 and give sampler_b to newly starting rollouts.
sampler_b = training_client.save_weights_and_get_sampling_client(kind="inflight")
# sampler_b requires W2 or newer and has a fresh KV cache namespace.
# sampler_a keeps its namespace and moves to W2 as it propagates.
# Until then, sampler_a can still be served by a replica on W1.
In a training loop, save with kind="inflight" after each optimizer step and hand the new client to the rollouts that start afterwards. Rollouts already in progress can keep their existing client. The async variant, save_weights_and_get_sampling_client_async, takes the same argument, and inflight saves work for both LoRA and full fine-tuned models.
Guarantees
- Weights are at least as new as the save. Samples from an inflight client always run on the weights saved when the client was created, or on newer ones.
- Saves return before every replica has loaded the weights. An inflight save completes once the checkpoint is saved and the shared weights are updated. Serving replicas load the update asynchronously, so a client created earlier may keep sampling from older weights for a short time.
- Each client has its own KV cache. Requests through the same client, including copies of it pickled and sent to other processes, can reuse cached prompt prefixes. Different inflight clients never share cache entries, and every save creates a new cache namespace, even if no optimizer step happened between saves.
- A generation can span an update. Publishing new weights does not clear an existing client's cache. A single generation can continue across an update, with earlier tokens and their KV computed under older weights. As a result, outputs can depend on when updates land, even with the same prompt and seed.
Training on inflight samples
Responses do not report which saved weights produced each token, and one sequence can mix tokens from several versions. Each token's logprob is still the logprob under the distribution it was actually sampled from. Treat inflight samples as off-policy data, and use a loss that corrects for the gap between the sampling and training policies. StableReinforce is built for highly off-policy async RL; create the client with both kind="inflight" and record_stability_info=True. Importance sampling, PPO, and CISPO also work, using the sampled logprobs.
Checkpoints
Like other sampling clients created by save_weights_and_get_sampling_client, inflight saves are ephemeral, and Tinker deletes older ones once newer weights have been published. To keep a particular set of weights, save it separately with save_weights_for_sampler.