Skip to content
View as Markdown

tinker.types.SampleResponse

Generated from tinker 0.39.0 at commit 4801e02. Source links point at that snapshot.

class tinker.types.SampleResponse()

Response from a sampling request.

Contains generated sequences and optional prompt-level log probabilities. Numpy fields provide direct array access without format conversion. The corresponding Python-list properties convert lazily on first access.

Fields:

  • sequences (Sequence[SampledSequence]) – Generated sequences. Each contains token IDs, optional logprobs, and stop reason.
  • prompt_logprobs_np (Optional[np.ndarray]) – Per-token log probabilities for the prompt as a 1-D float32 numpy array, shape (prompt_length,). NaN at positions where logprobs were not computed (e.g. the first prompt token). None if prompt logprobs were not requested.
  • topk_prompt_logprobs_np (Optional[TopkLogprobs]) – Top-k prompt logprobs as a pair of dense matrices (see TopkLogprobs). None if top-k was not requested.
  • target_prompt_logprobs (Optional[TensorData]) – Logprobs of the ids in SampleRequest.target_prompt_logprobs: a float32 tensor of the same shape and layout, 0.0 where the request had -1. None if not requested.

  • prompt_cache_hit_tokens (int) – Number of prompt tokens billed as prefix-cache hits.

    Counted on the prompt itself: for num_samples > 1 the prompt is shared, so this is not multiplied across samples. Prefill on the shared prompts samples (the remaining num_samples - 1) is billed as cache hits.

  • prompt_alt_tokens (Optional[PromptAltTokens]) – The draws SampleRequest.prompt_alt_tokens_k asked for: k token ids and their logprobs at every prompt position after the first, as two [len(prompt) - 1, k] tensors (see PromptAltTokens). None if not requested.

prompt_logprobs()

Per-token log probabilities for the prompt as a Python list.

If prompt_logprobs was set to true in the request, logprobs are computed for every token in the prompt, or for the last SampleRequest.prompt_logprobs_last_n tokens if set. Each entry is a float, or None for positions where logprobs were not computed (e.g. the first prompt token). Returns None if prompt logprobs were not requested.

Converted from prompt_logprobs_np on first access (cached afterwards).

Returns: Optional[List[Optional[float]]]

topk_prompt_logprobs()

Top-k prompt logprobs as nested Python lists.

If topk_prompt_logprobs was set to a positive integer k in the request, the top-k logprobs are computed for every token in the prompt, or for the last SampleRequest.prompt_logprobs_last_n tokens if set. For each prompt position: a list of up to k (token_id, logprob) tuples, or None for positions where logprobs were not computed. Returns None if top-k was not requested.

Converted from topk_prompt_logprobs_np on first access (cached afterwards).

Returns: Optional[list[Optional[list[tuple[int, float]]]]]

Referenced by