# GSPO

GSPO ([Zheng et al., 2025](https://arxiv.org/abs/2507.18071)) replaces PPO's token-level importance ratio with a sequence-level one: the geometric mean of the per-token ratios (\\frac{p\_\\theta(x_t)}{q(x_t)}) over a sequence's sampled tokens. Clipping that one ratio keeps or drops whole sequences, instead of clipping individual tokens.

Tinker implements the token-level variant (GSPO-token). Every token carries the sequence ratio's value, but the gradient of its own log-probability, so per-token advantages still work.

The sequence ratio, over the tokens (M) that `sequence_ratio_mask` marks, is:

[ s(\\theta) = \\exp\\left(\\frac{1}{|M|} \\sum\_{t \\in M} \\log \\frac{p\_\\theta(x_t)}{q(x_t)}\\right) ]

Each token's ratio takes (s)'s value and its own gradient:

\[ s_t(\\theta) = \\textbf{sg}\\left[s(\\theta)\\right] \\cdot \\frac{p\_\\theta(x_t)}{\\textbf{sg}\\left[p\_\\theta(x_t)\\right]} \]

The GSPO objective is:

\[ \\mathcal{L}_{\\text{GSPO}}(\\theta) = -\\sum_t \\min\\left(s_t(\\theta) \\cdot A_t, \\text{clip}\\left(s_t(\\theta), 1-\\epsilon_{\\text{low}}, 1+\\epsilon\_{\\text{high}}\\right) \\cdot A_t\\right) \]

This is implemented as:

```python
# Sequence log-ratio: the mean token log-ratio over the masked tokens
in_ratio = sequence_ratio_mask != 0
log_ratio = (target_logprobs.detach() - ref_logprobs)[in_ratio].mean()
# Every token carries the sequence ratio's value with its own log-probability's gradient
ratio = torch.exp(log_ratio + target_logprobs - target_logprobs.detach())
# Apply clipping
clipped_ratio = torch.clamp(ratio, clip_low_threshold, clip_high_threshold)
# Take minimum (most conservative), as in PPO
gspo_objective = torch.minimum(ratio * advantages, clipped_ratio * advantages)
# GSPO loss is negative of objective
loss = -gspo_objective.sum()
```

**Input tensors:**

- `target_tokens: array[(N,), int]` — Target token IDs (from the sampler (q))
- `logprobs: array[(N,), float]` — `ref_logprobs` for the tokens
- `advantages: array[(N,), float]` — Advantage values for RL
- `sequence_ratio_mask: array[(N,), float]` — Nonzero on the tokens the sequence ratio, and so the clip, is taken over: typically the sampled completion tokens. It must mark at least one token.

**Output tensors:**

- `logprobs: array[(N,), float]` — `target_logprobs` for the tokens

**Output diagnostics:**

- `loss:sum` (scalar) — Sum of GSPO losses
- `gspo_sequence_log_ratio:mean` (scalar) — Mean over sequences of (\\log s(\\theta))
- `gspo_clipped_fraction:mean` (scalar) — Fraction of sequences whose ratio fell outside the clip range

## Building a datum

To take both the ratio and the loss over only the sampled tokens:

```python
import numpy as np

n_prompt = len(prompt_tokens) - 1  # targets are shifted by one
n = len(target_tokens)
completion = np.arange(n) >= n_prompt

datum = types.Datum(
    model_input=types.ModelInput.from_ints(tokens[:-1]),
    loss_fn_inputs={
        "target_tokens": np.asarray(target_tokens, dtype=np.int64),
        "logprobs": np.asarray(logprobs, dtype=np.float32),
        "advantages": np.where(completion, advantage, 0.0).astype(np.float32),
        "sequence_ratio_mask": completion.astype(np.float32),
    },
)
```

> **SDK versions**
>
> With tinker SDK 0.33.1 and earlier, pass `sequence_ratio_mask` as a NumPy array or torch tensor; those versions reject it as a plain Python list.

## Choosing clipping thresholds

Because the clip applies to the sequence ratio, and a geometric mean moves far less than a single token's ratio, GSPO's clip range is much narrower than PPO's. Tinker defaults to the paper's range, `clip_low_threshold=1 - 3e-4` and `clip_high_threshold=1 + 4e-4`. To set it explicitly:

```python
fwd_bwd_future = await training_client.forward_backward_async(
    data=data,
    loss_fn="gspo",
    loss_fn_config={"clip_low_threshold": 1 - 3e-4, "clip_high_threshold": 1 + 4e-4}
)
fwd_bwd_result = await fwd_bwd_future.result_async()
```