View as Markdown

GSPO

GSPO (Zheng et al., 2025) replaces PPO's token-level importance ratio with a sequence-level one: the geometric mean of the per-token ratios \(\frac{p_\theta(x_t)}{q(x_t)}\) over a sequence's sampled tokens. Clipping that one ratio keeps or drops whole sequences, instead of clipping individual tokens.

Tinker implements the token-level variant (GSPO-token). Every token carries the sequence ratio's value, but the gradient of its own log-probability, so per-token advantages still work.

The sequence ratio, over the tokens \(M\) that sequence_ratio_mask marks, is:

\[ s(\theta) = \exp\left(\frac{1}{|M|} \sum_{t \in M} \log \frac{p_\theta(x_t)}{q(x_t)}\right) \]

Each token's ratio takes \(s\)'s value and its own gradient:

\[ s_t(\theta) = \textbf{sg}\left[s(\theta)\right] \cdot \frac{p_\theta(x_t)}{\textbf{sg}\left[p_\theta(x_t)\right]} \]

The GSPO objective is:

\[ \mathcal{L}_{\text{GSPO}}(\theta) = -\sum_t \min\left(s_t(\theta) \cdot A_t, \text{clip}\left(s_t(\theta), 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}\right) \cdot A_t\right) \]

This is implemented as:

# Sequence log-ratio: the mean token log-ratio over the masked tokens
in_ratio = sequence_ratio_mask != 0
log_ratio = (target_logprobs.detach() - ref_logprobs)[in_ratio].mean()
# Every token carries the sequence ratio's value with its own log-probability's gradient
ratio = torch.exp(log_ratio + target_logprobs - target_logprobs.detach())
# Apply clipping
clipped_ratio = torch.clamp(ratio, clip_low_threshold, clip_high_threshold)
# Take minimum (most conservative), as in PPO
gspo_objective = torch.minimum(ratio * advantages, clipped_ratio * advantages)
# GSPO loss is negative of objective
loss = -gspo_objective.sum()

Input tensors:

target_tokens: array[(N,), int] — Target token IDs (from the sampler \(q\))
logprobs: array[(N,), float] — ref_logprobs for the tokens
advantages: array[(N,), float] — Advantage values for RL
sequence_ratio_mask: array[(N,), float] — Nonzero on the tokens the sequence ratio, and so the clip, is taken over: typically the sampled completion tokens. It must mark at least one token.

Output tensors:

logprobs: array[(N,), float] — target_logprobs for the tokens

Output diagnostics:

loss:sum (scalar) — Sum of GSPO losses
gspo_sequence_log_ratio:mean (scalar) — Mean over sequences of \(\log s(\theta)\)
gspo_clipped_fraction:mean (scalar) — Fraction of sequences whose ratio fell outside the clip range

Building a datum

To take both the ratio and the loss over only the sampled tokens:

import numpy as np

n_prompt = len(prompt_tokens) - 1  # targets are shifted by one
n = len(target_tokens)
completion = np.arange(n) >= n_prompt

datum = types.Datum(
    model_input=types.ModelInput.from_ints(tokens[:-1]),
    loss_fn_inputs={
        "target_tokens": np.asarray(target_tokens, dtype=np.int64),
        "logprobs": np.asarray(logprobs, dtype=np.float32),
        "advantages": np.where(completion, advantage, 0.0).astype(np.float32),
        "sequence_ratio_mask": completion.astype(np.float32),
    },
)

SDK versions

With tinker SDK 0.33.1 and earlier, pass sequence_ratio_mask as a NumPy array or torch tensor; those versions reject it as a plain Python list.

Choosing clipping thresholds

Because the clip applies to the sequence ratio, and a geometric mean moves far less than a single token's ratio, GSPO's clip range is much narrower than PPO's. Tinker defaults to the paper's range, clip_low_threshold=1 - 3e-4 and clip_high_threshold=1 + 4e-4. To set it explicitly:

fwd_bwd_future = await training_client.forward_backward_async(
    data=data,
    loss_fn="gspo",
    loss_fn_config={"clip_low_threshold": 1 - 3e-4, "clip_high_threshold": 1 + 4e-4}
)
fwd_bwd_result = await fwd_bwd_future.result_async()