Experimental: call RestClient.search_checkpoints(CheckpointSearchRequest(...)) to list matching checkpoints newest first. Filter by project_ids, user_metadata, checkpoint_type, owner, or base_models, and page with next_cursor. Results are limited to projects you can list, and the default owner filter selects your own checkpoints.
Read Checkpoint.base_model to get the base model a checkpoint was trained from. It is None when the server does not send it.
Use APIFuture.future() when passing an operation to concurrent.futures.wait, as_completed, or callbacks. Use .coroutine() with APIs requiring a coroutine, for example asyncio.create_task(client.sample(...).coroutine()).
ServiceClient.create_lora_training_client(), create_training_client_from_state(), create_training_client_from_state_with_optimizer(), create_sampling_client(), TrainingClient.create_sampling_client(), and save_weights_and_get_sampling_client() return futures of clients. Resolve them with await or .result() before using the clients; using the compatibility wrapper directly still works with a deprecation warning.
Raises RuntimeError when a blocking SDK call, such as .result() on a pending future, runs on the SDK's own event loop, for example inside a done-callback. Before, the call logged a warning and never returned.
Points deprecation warnings at the user call site instead of SDK wrappers.
Deprecations
The _async methods on service, training, sampling, REST, and admin clients are deprecated. Await the corresponding unsuffixed method instead. For training operations that previously required two awaits, replace f = await client.forward_backward_async(...); result = await f.result_async() with result = await client.forward_backward(...). No removal version or date is specified.
Using a client-creation future directly as a training or sampling client is deprecated. Add .result() in synchronous code or await in asynchronous code; the compatibility behavior remains available in this release, with no removal version or date specified.
Calling standard concurrent.futures.Future methods directly on sampling futures is deprecated. Use .future() first, for example client.sample(...).future().add_done_callback(callback). Sampling compatibility remains available in this release.
REST and admin operations now return awaitable APIFuture objects instead of concurrent.futures.Future objects. Callers using .done(), .add_done_callback(), wait, or as_completed must first call .future(): replace rest_client.list_sessions().add_done_callback(cb) with rest_client.list_sessions().future().add_done_callback(cb). Plain .result() callers continue to work.
The experimental RestClient.search_sessions_async() method is removed. Replace await rest_client.search_sessions_async(request) with await rest_client.search_sessions(request).
The client-creation methods return before the client exists. An error such as a bad checkpoint path now raises at .result(), at await, or at the first call on the client, not inside the creation call. To handle the error at creation, call .result() inside your try block: training_client = service_client.create_training_client_from_state(path).result().
Call save_weights_and_get_sampling_client(kind="inflight") to publish mutable weights shared by that training model's inflight clients. Each returned client has a fresh KV-cache namespace and uses the just-published weights or newer; existing clients adopt updates asynchronously. Active generations can span updates. The default kind="snapshot" still keeps fixed weights.
Read StorageBillingEvent.free to distinguish free storage from paid storage in billing usage responses. Older responses without the field default to False.
Changes
forward_backward_custom() returns its future at once. It no longer waits for the forward pass and the local loss. Requests still run in submission order, so you can call optim_step() right after it.
Capacity-pause messages are emitted only after the pause has lasted 15 minutes, reducing warnings for short periods spent waiting for capacity.
An expired JWT credential, from api_key, TINKER_API_KEY or TINKER_CREDENTIAL_CMD, now raises a local TinkerError, and the SDK sends no request. Before, the server rejected it with AuthenticationError. A JWT credential without an exp claim raises ValueError.
Fixed
Prevents a slow custom loss from delaying session heartbeats and other API calls.
Recovers when the server rejects the SDK's short-lived session token as expired or signed with a retired key: the SDK gets a new token and sends the request again once. Concurrent rejections share one new token.
Surfaces credential rejection during token exchange instead of continuing to use a cached JWT.
Refreshes the session token in time when the local clock runs behind the server's. Before, a clock more than five minutes behind made requests fail with AuthenticationError near the end of each token's lifetime.
Deprecations
None.
Breaking changes
After a client-side custom loss fails or is cancelled, the TrainingClient rejects all queued and later requests with RuntimeError. Create a new training client, for example from your last checkpoint, before you continue.
Alpha: call ServiceClient.create_custom_loss(fn, input_ranks=...).result() to create a session-scoped LossHandle, then pass it to TrainingClient.forward_backward_custom(). The function receives logprobs and named tensor inputs per datum and returns a scalar loss plus scalar metrics. It must satisfy the documented torch tracing constraints. When the server enables it for your organization, the loss runs on the server in one request; otherwise it runs locally.
Use tinker.AdminClient() for get_billing_usage(), get_current_checkpoint_storage_usage(), and get_audit_log(). get_billing_usage() and get_current_checkpoint_storage_usage() require the admin or billing-admin role; get_audit_log() requires the admin role.
Experimental: search sessions with RestClient.search_sessions(SessionSearchRequest(...)), using project, owner, metadata, display-name, and status filters. Page through results with next_cursor. Experimental APIs emit tinker.ExperimentalWarning and may change without a deprecation period.
Billing, current-storage, and audit-log methods on RestClient are deprecated in favor of the corresponding AdminClient methods. Their async variants are also deprecated on RestClient; no removal version or date is specified.
The loss_type_input parameter of forward_backward_custom() and its async variant is deprecated because custom losses always receive logprobs. Remove the argument; no removal version or date is specified.
tinker checkpoint download and tinker checkpoint push-hf no longer accept sampler checkpoint paths (tinker://<run-id>/sampler_weights/<name>). Pass a training checkpoint (tinker://<run-id>/weights/<name>) or an external weights path (tinker://<run-id>/external_weights/<name>). A training checkpoint download starts a temporary training session, exports the weights, and deletes the export afterwards. Encrypted files keep .age; decrypt encrypted adapters before Hugging Face upload.
Use loss_fn="stable_reinforce" for StableReinforce training. Create sampling clients with record_stability_info=True when recording the sequence stability information required by that workflow; the recorded information is session-scoped and expires after inactivity.
Import the shared provenance-span type as tinker.types.ProvenanceSpans when constructing or annotating datum provenance.
Changes
None.
Fixed
Accepts zero-length spans in PromptProvenanceSpan and SampledProvenanceSpan. A one-token generation, whose sampled run in the model input is empty, no longer needs a special case.
Pass loss_fn="gspo" to training calls; sequence_ratio_mask is recognized as a float32 loss input.
Set ServiceClient(display_name=...) to give the session a human-readable name. Session responses include that name; it need not be unique.
Run tinker model list [substring] to list the models available to your organization, including context limits and whether each model can train or sample. The optional name filter is case-insensitive.
Run tinker auth whoami to display the active credential's email, user URN, and organization URN. RestClient.whoami().result().org_urn also returns the organization URN.
Server-sent deprecation notices are now logged at the warning or error level indicated by the response.
Fixed
Sends one idempotency key for all retries of a copy_weights() call, so a retry does not create a second copy.
Excludes cross-account checkpoint-copy credentials from debug logs.
Deprecations
None.
Breaking changes
ServiceClient.copy_weights() now delegates to the REST copy API and creates a new session for the copy instead of using the client's current session. Callers must provide a destination project, either through the service client's project_id, the REST method's destination_project_id, or TINKER_PROJECT_ID; missing project selection raises ValueError. Update code that assumed the copy belonged to the existing session.
RestClient.get_checkpoint_archive_url*() is deprecated, including async and Tinker-path variants. Export with TrainingClient.save_weights_external() and fetch per-file URLs with RestClient.get_external_weights_urls() instead. These methods remain in this release, although the server may already reject archive downloads for some organizations. The notice specifies a future SDK removal without a version or date.
Use prompt_logprobs_last_n=N with SamplingClient.sample(..., include_prompt_logprobs=True) or compute_logprobs(...) to score the last N prompt tokens. Earlier entries are None; N must be between 1 and len(prompt) - 1. This also limits top-k prompt logprobs, but does not affect target-token scoring or alternative prompt-token draws.
Pass age_encryption_recipients=[...] to TrainingClient.save_weights_external() to encrypt each exported file to up to 10 age or SSH recipients. Exported files have a .age suffix; decrypt them with age -d -i <identity>. Omitting the argument keeps exports unencrypted.
Call RestClient.get_current_checkpoint_storage_usage(project_id=...) or tinker billing checkpoint-storage --project-id ... --csv ... to report checkpoint counts, sizes, and projected costs grouped by project and owner. The project filter is optional; data may lag by 1–2 hours, and costs are gross 720-hour projections rather than invoices.
Custom losses can consume additional tensors in Datum.loss_fn_inputs, which previously raised ValueError. Only target_tokens and weights are sent for the server forward pass; other fields remain available to the local loss function.
Changes
Copies of a SamplingClient in other processes get request IDs from the server, so their requests never collide.
Fixed
Sends requests from an unpickled SamplingClient to the parent's server when the parent set it with TINKER_BASE_URL. Before, a worker without that variable used the default server.
Raises an error on the first request when TINKER_CREDENTIAL_CMD fails or prints nothing. Before, the SDK retried it as a connection error and raised APIConnectionError after about six minutes.
Deprecations
None.
Breaking changes
Callers passing HTTP-client options through ServiceClient must migrate: options previously forwarded through **kwargs are now ignored with FutureWarning. Use the supported base_url, api_key, and default_headers_overrides arguments. For custom headers, replace ServiceClient(default_headers={"X-Example": "value"}) with ServiceClient(default_headers_overrides={"X-Example": "value"}). Remove other unsupported options and do not rely on them taking effect. There is no replacement for timeout, max_retries, default_query or http_client; if you need them, stay on 0.31.0. A future SDK release makes these arguments an error; no version or date is set.
Set prompt_alt_tokens_k=k on SamplingClient.sample() to draw alternatives at each prompt position after the first. Read SampleResponse.prompt_alt_tokens.tokens and .logprobs; both have shape [len(prompt) - 1, k], and row i describes prompt position i + 1. Draws use the request's temperature.
Changes
Estimated usage costs returned by RestClient.get_billing_usage() and tinker billing usage are priced independently for each model and usage category on completed days, so one unreconciled group does not leave the whole day unpriced. cost_data_through separates invoice-derived estimates from published-rate estimates; unavailable costs can still be null on either side. These are usage estimates, not invoice amounts due.
Fixed
Tolerates unknown billing event fields during strict parsing while preserving known fields.
Use the restored public DimuonOptimizerConfig and DimuonParams to configure Dimuon requests in the SDK. Execution requires server enablement.
Changes
None.
Fixed
None.
Deprecations
None.
Breaking changes
When upgrading Dimuon code from 0.30.2 or earlier, remove damping and rank_lr_correction_exponent from DimuonParams(...); these arguments are no longer accepted.
DimuonOptimizerConfig and DimuonParams are removed from the public API. If you use Dimuon, remain on 0.30.2 until a release restores public support. Adam-based training continues to work.
Use RestClient.get_billing_usage() or tinker billing usage --csv to read estimated USD usage costs and effective token or storage rates. cost_data_through marks the end of invoice-reconciled cost coverage. Estimates can be unavailable and exclude credits and committed spend; they are not invoice amounts due. Requires billing-view access in your organization.
Changes
None.
Fixed
Rejects malformed sparse-CSR indices when converting TensorData to PyTorch tensors.
Reports the custom loss in forward_backward_custom() metrics under loss:sum, rather than the surrogate loss. The previous value is available under surrogate_loss:sum; this changes metrics recorded by custom-loss training runs.
Suppresses PyTorch’s sparse-CSR beta warning during tensor conversions.
Pass target_prompt_logprobs to SamplingClient.sample() to score selected token IDs at prompt positions. Supply an int64TensorData with shape [len(prompt) - 1, K], using -1 for unused cells, and read SampleResponse.target_prompt_logprobs.
Use an integer pad_value with TensorData.from_torch_sparse(), .to_numpy(), .to_torch(), or .tolist() to represent omitted sparse entries with a value other than zero, such as -1.
Use RestClient.get_session(session_id) to retrieve the session’s custom metadata in GetSessionResponse.user_metadata.
Use with tinker.ServiceClient() as client: to close a session automatically. Normal completion marks it successful, an exception marks it errored, and an interruption marks it interrupted.
Use optimizer=types.DimuonOptimizerConfig() and optim_step(types.DimuonParams(learning_rate=...)) to configure the SDK’s Dimuon requests. Execution requires server enablement. Existing Adam calls, including optim_step(adam_params=...), still work.
Changes
None.
Fixed
Points SaveWeightsForSamplerResponse.get_console_url() at the saved checkpoint instead of a malformed checkpoint URL.
Deprecations
None.
Breaking changes
Checkpoint expiration now has a maximum of 10 years. If you pass a larger ttl_seconds when saving or copying weights, reduce it to at most 315360000, or use None for a checkpoint that never expires.
Pass topk_sample_logprobs=k to SamplingClient.sample() to return the top token alternatives at each generated position. Read each sequence’s .topk_logprobs as token-ID/log-probability pairs or .topk_logprobs_np as arrays.
Changes
None.
Fixed
None.
Deprecations
None.
Breaking changes
tinker auth login now opens the console’s API-key creation page and prompts you to paste a key. The --api-key flag is removed; replace tinker auth login --api-key with tinker auth login.
RestClient.get_external_weights_urls() now accepts one Tinker path instead of a training-run ID and checkpoint ID. Replace rest_client.get_external_weights_urls(run_id, "external_weights/name") with rest_client.get_external_weights_urls(f"tinker://{run_id}/external_weights/name"), or pass the .path returned by save_weights_external(...).result(). Apply the same change to the async method.
Call TrainingClient.save_weights_external(name) to request an external-weight checkpoint. The SDK adds RestClient.get_external_weights_urls(training_run_id, checkpoint_id) for its per-file signed download URLs; the checkpoint ID has the form "external_weights/<name>". External-weight downloads require server enablement for your organization.
Call ServiceClient.close(status, detail=None) to finish a session with status "success", "errored", or "interrupted". Await the returned future or call .result() before exiting.
Changes
None.
Fixed
Prevents pending-task warnings when a sampling client is garbage-collected.
Pass user_metadata to save_state() or save_weights_for_sampler() to attach custom metadata to a checkpoint. Overwriting a state checkpoint replaces its metadata; omitting user_metadata clears the previous values.
Call get_console_url() on service and training clients or checkpoint-save responses to open them in the console. Call get_playground_url() on a sampler-checkpoint response to open the saved model in the playground.
Pass stop=[] in SamplingParams to disable implicit end-of-sequence stopping and sample until max_tokens.
Subprocess-isolated sampling is removed. Remove subprocess_sampling= from direct SamplingClient construction; TINKER_SUBPROCESS_SAMPLING no longer has an effect. Move CPU-heavy grading or environment work into your own process or thread pool if it would block sampling.
Pass string values as well as numbers in loss_fn_config when calling forward() or forward_backward().
Call ServiceClient.copy_weights() to copy a checkpoint into the client’s project. Await the returned future, or call .result(), to get the new checkpoint path.
Use RestClient.get_audit_log(event_type="projects"), "teams", or "organizations" to retrieve administrative audit events. Read AuditLogEntry.event_details for their full details.
Use Checkpoint.user_metadata to read custom metadata returned with a checkpoint.
Changes
Validates manually entered API keys before saving them and shows their account and organization.
Fixed
Clears locally stored credentials during logout even if server-side key revocation fails, and reports the key for manual revocation.
Deprecations
AuditLogEntry.model_id, .tinker_path, and .purpose are deprecated. Read the corresponding values from .event_details instead. No removal version or date is announced.
Use tinker auth login to sign in through a browser and save credentials for future SDK and CLI sessions. Use tinker auth status to check access and tinker auth logout to sign out.
Call tinker.get_tinker_token() to retrieve a static API key, tinker.tinker_has_credentials() to check whether credentials are configured, or tinker.raise_if_tinker_not_accessible() to check server access. Payment-required responses raise BillingError.
Use Datum.with_provenance() with PromptProvenanceSpan and SampledProvenanceSpan to identify the prompt and sampled tokens in training data. Sampling results include SampledSequence.sequence_id for linking sampled spans.
Use SupportedModel.trainable and .sampleable from server capabilities to check which operations a model supports. Either field can be None when the server does not report it.
Changes
None.
Fixed
None.
Deprecations
None.
Breaking changes
Code that constructs SampledSequence directly must now pass sequence_id along with stop_reason, tokens_np, and logprobs_np. When reconstructing a sampling result, preserve its returned sequence ID.
Preserves each example's tensor shape and dtype in forward/backward results when loss outputs have different shapes or dtypes. Previously, later examples could use the first example's shape and dtype.
Deprecations
None.
Breaking changes
The checkpoint restore methods add base_model as the second positional argument, so callers must pass metadata and credentials by keyword: client.create_training_client_from_state(path, user_metadata=metadata, weights_access_token=token). This also applies to the optimizer and async variants. Use base_model=... to choose a checkpoint-compatible model variant, such as one with a different context length.
Run tinker session export-trace SESSION_ID to download a session's training and sampling timeline as a Perfetto trace. Use --output PATH for a custom file or --url-only to print its download URL. The SDK exposes the same export through RestClient.export_session_trace(session_id).
Changes
Updates the tokenizer revision used for Kimi-K2.5.
Call RestClient.get_billing_usage(starting_on, ending_before) or run tinker billing usage STARTING_ON ENDING_BEFORE to inspect hourly usage. Requires billing view access in your organization. Use UTC hour boundaries no more than 14 days apart; results report quantities rather than dollar charges. The CLI supports --csv PATH for usage rows and --sessions-csv PATH for session metadata.
Call RestClient.whoami().result() to get the credential's user URN and email; service identities can have no email. JWT authentication must be enabled or the credential must itself be a JWT.
Changes
Creates sessions only when a training or sampling client needs one. Constructing ServiceClient() and making REST-only calls no longer create a session, allowing read-only account access without session-creation permissions.
Use DmelChunk(dmel=...) to include serialized DMel tensors in ModelInput. Its length property reads the token count from the tensor.
Read SampleResponse.prompt_cache_hit_tokens for the number of prompt tokens billed as prefix-cache hits. The count refers to the shared prompt and is not multiplied by num_samples.
Set TINKER_PROJECT_ID to select the project used by ServiceClient() when no project_id argument is supplied.
Changes
None.
Fixed
None.
Deprecations
None.
Breaking changes
Sampling concurrency now follows the server-provided limit, overriding RetryConfig.max_connections. If your application needs a lower cap, limit concurrent sampling calls in the application.
Pass RetryConfig(enable_stuck_detection=False) when creating a sampling client to disable the no-progress timeout. Requests can then wait indefinitely for progress.
Changes
None.
Fixed
Accepts explicit checkpoint paths in tinker checkpoint delete -y without an argument-handling error, and restores tinker checkpoint push-hf upload filters.