Tutorial 103: Efficient Sampling with Tinker
Prerequisites
Run it interactively [source]
Tinker runs on remote GPUs. Every API call involves network latency plus GPU compute time. If you send sampling requests one at a time -- send, wait, send, wait -- you spend most of your time idle while Tinker works.
The solution: submit requests concurrently, then collect their results with asyncio.gather. Tinker can batch and pipeline concurrent requests on the GPU, reducing total elapsed time compared with sending them one at a time. This matters most for sampling, where RL training may require hundreds of completions per step.
SDK request methods use one interface: they return an APIFuture immediately, without an _async suffix. Calling sample() submits a request; await on its future yields the finished SampleResponse. Client creation works the same way: await create_sampling_client() before using the client. There is no separate submission await or second result await. For synchronous code, the same future has a blocking .result() method; avoid it inside an async function.
import asyncio
import os
import time
import warnings
warnings.filterwarnings("ignore", message="IProgress not found")
import tinker
from tinker_cookbook.renderers import get_renderer, get_text_content
Setup
We create a SamplingClient for the base Qwen3.5-4B model (no fine-tuning needed for this tutorial). We also set up a renderer to handle the chat template and a list of diverse prompts to sample from.
mo.stop(
"TINKER_API_KEY" not in os.environ and not api_key.value,
"Paste your API key above",
)
if api_key.value:
os.environ["TINKER_API_KEY"] = api_key.value
BASE_MODEL = "Qwen/Qwen3.5-4B"
service_client = tinker.ServiceClient()
sampling_client = await service_client.create_sampling_client(base_model=BASE_MODEL)
tokenizer = sampling_client.get_tokenizer()
renderer = get_renderer("qwen3_5", tokenizer)
stop_sequences = renderer.get_stop_sequences()
params = tinker.SamplingParams(max_tokens=150, temperature=0.7, stop=stop_sequences)
# A diverse set of prompts to sample from
prompts = [
"What causes thunder?",
"Write a haiku about the ocean.",
"What is the capital of New Zealand?",
"Explain what a hash table is in two sentences.",
"Name three inventions from the 19th century.",
"Why do leaves change color in autumn?",
"Translate to Spanish: The library closes at nine.",
"What is the smallest prime number greater than 50?",
]
print(f"Model: {BASE_MODEL}")
print(f"Prompts: {len(prompts)}")
Sequential sampling (the slow way)
The simplest approach: for each prompt, build the generation input, call sample(), and immediately await its future before moving on to the next. This suspends the current coroutine without blocking the event loop, but the loop still sends requests sequentially: each request waits for the previous one to complete before starting.
_start = time.time()
sequential_results = []
for _prompt_text in prompts:
_messages = [{"role": "user", "content": _prompt_text}]
_model_input = renderer.build_generation_prompt(_messages)
_result = await sampling_client.sample(
prompt=_model_input, num_samples=1, sampling_params=params
)
_response_msg, _ = renderer.parse_response(_result.sequences[0].tokens)
sequential_results.append(
get_text_content(_response_msg)
) # Wait for each request before sending the next
sequential_time = time.time() - _start
for _prompt_text, _answer in zip(prompts, sequential_results):
print(f"Q: {_prompt_text}")
print(f"A: {_answer[:120]}...\n")
print(
f"Sequential: {sequential_time:.1f}s for {len(prompts)} prompts ({sequential_time / len(prompts):.1f}s each)"
)
Output
Q: What causes thunder?
A: Here's a thinking process that leads to the explanation of thunder's cause:
1. **Analyze the Request:**
* **Ques...
Q: Write a haiku about the ocean.
A: Thinking Process:
1. **Analyze the Request:**
* Topic: The ocean.
* Form: Haiku (5-7-5 syllables).
2. **...
Q: What is the capital of New Zealand?
A: Thinking Process:
1. **Identify the core question:** The user is asking for the capital of New Zealand.
2. **Retrieve...
Q: Explain what a hash table is in two sentences.
A: Thinking Process:
1. **Analyze the Request:**
* Topic: Hash table.
* Constraint: Exactly two sentences.
...
Q: Name three inventions from the 19th century.
A: Thinking Process:
1. **Analyze the Request:**
* Task: Name three inventions.
* Constraint: They must be fr...
Q: Why do leaves change color in autumn?
A: Here's a thinking process that leads to the explanation of why leaves change color in autumn:
1. **Deconstruct the Req...
Q: Translate to Spanish: The library closes at nine.
A: We need to translate "The library closes at nine." into Spanish. Let's break it down.
First, "The library" = "La biblio...
Q: What is the smallest prime number greater than 50?
A: Thinking Process:
1. **Analyze the Request:** The user is asking for the smallest prime number that is greater than 50...
Sequential: 14.2s for 8 prompts (1.8s each)
Concurrent sampling with asyncio.gather
Each sample() call submits work immediately and returns an APIFuture. Build a list of those futures without awaiting inside the submission loop, then pass them to asyncio.gather to await all results. The key insight: submit all requests first, then collect results. Tinker batches concurrent requests on the GPU for higher throughput.
_start = time.time()
# Step 1: Submit ALL requests before awaiting any results
_futures = []
for _prompt_text in prompts:
_messages = [{"role": "user", "content": _prompt_text}]
_model_input = renderer.build_generation_prompt(_messages)
_futures.append(sampling_client.sample(
prompt=_model_input, num_samples=1, sampling_params=params
))
_results = await asyncio.gather(*_futures)
concurrent_results = []
for _result in _results:
# Step 2: Parse results (all requests were running in parallel)
_response_msg, _ = renderer.parse_response(_result.sequences[0].tokens)
concurrent_results.append(get_text_content(_response_msg))
concurrent_time = time.time() - _start
for _prompt_text, _answer in zip(prompts, concurrent_results):
print(f"Q: {_prompt_text}")
print(f"A: {_answer[:120]}...\n")
print(f"Concurrent: {concurrent_time:.1f}s for {len(prompts)} prompts")
print(f"Sequential: {sequential_time:.1f}s for {len(prompts)} prompts")
print(f"Speedup: {sequential_time / concurrent_time:.1f}x")
Output
Q: What causes thunder?
A: Here's a thinking process that leads to the explanation of thunder:
1. **Analyze the Request:**
* **Question:** ...
Q: Write a haiku about the ocean.
A: Thinking Process:
1. **Analyze the Request:**
* Topic: The ocean.
* Form: Haiku (5-7-5 syllables).
2. **...
Q: What is the capital of New Zealand?
A: The capital of New Zealand is **Wellington**....
Q: Explain what a hash table is in two sentences.
A: Thinking Process:
1. **Analyze the Request:**
* Topic: Hash table.
* Constraint: Exactly two sentences.
...
Q: Name three inventions from the 19th century.
A: Thinking Process:
1. **Analyze the Request:**
* Task: Name three inventions.
* Constraint: They must be fr...
Q: Why do leaves change color in autumn?
A: Here's a thinking process that leads to the explanation of why leaves change color in autumn:
1. **Analyze the Request...
Q: Translate to Spanish: The library closes at nine.
A: We need to translate the sentence "The library closes at nine." into Spanish. Let's break it down.
First, identify the ...
Q: What is the smallest prime number greater than 50?
A: Thinking Process:
1. **Identify the goal:** The user wants to find the smallest prime number greater than 50.
2. **D...
Concurrent: 2.2s for 8 prompts
Sequential: 14.2s for 8 prompts
Speedup: 6.5x
Multiple completions per prompt (num_samples)
In GRPO-style RL, you need group_size independent completions for each problem so you can compare them and compute advantages. The num_samples parameter generates multiple completions in a single API call -- more efficient than sending separate requests for the same prompt.
_GROUP_SIZE = 4
test_prompt = "Name a famous scientist and explain their key contribution in one sentence."
_messages = [{"role": "user", "content": test_prompt}]
_model_input = renderer.build_generation_prompt(_messages)
_start = time.time()
_result = await sampling_client.sample(
prompt=_model_input, num_samples=_GROUP_SIZE, sampling_params=params
)
# Single call with num_samples=4 -- generates 4 independent completions
multi_time = time.time() - _start
print(f"Prompt: {test_prompt}\n")
for i, _seq in enumerate(_result.sequences):
_response_msg, _ = renderer.parse_response(_seq.tokens)
text = get_text_content(_response_msg)
print(f"Completion {i + 1}: {text[:150]}\n")
_start = time.time()
for _ in range(_GROUP_SIZE):
await sampling_client.sample(
prompt=_model_input, num_samples=1, sampling_params=params
)
sequential_multi_time = time.time() - _start
print(f"num_samples={_GROUP_SIZE} in one call: {multi_time:.1f}s")
print(f"{_GROUP_SIZE} sequential calls: {sequential_multi_time:.1f}s")
# Compare: 4 sequential single calls
print(f"Speedup: {sequential_multi_time / multi_time:.1f}x")
Output
Prompt: Name a famous scientist and explain their key contribution in one sentence.
Completion 1: Thinking Process:
1. **Analyze the Request:**
* Task: Name a famous scientist and explain their key contribution.
* Constraint: Explain
Completion 2: Thinking Process:
1. **Analyze the Request:**
* Task: Name a famous scientist.
* Task: Explain their key contribution.
* Constrain
Completion 3: Thinking Process:
1. **Analyze the Request:**
* Target: Name a famous scientist.
* Task: Explain their key contribution.
* Constra
Completion 4: Thinking Process:
1. **Analyze the Request:**
* Task: Name a famous scientist and explain their key contribution.
* Constraint: Explain
num_samples=4 in one call: 1.9s
4 sequential calls: 6.6s
Speedup: 3.4x
Putting it together: batch evaluation
Combine both techniques -- concurrent requests across prompts and num_samples per prompt -- for maximum throughput. This is exactly the pattern used in RL training: submit many sampling requests in parallel, each generating a group of completions, then collect and grade them all.
_GROUP_SIZE = 4
_start = time.time()
# Submit all requests concurrently using asyncio.gather, each with num_samples=GROUP_SIZE
async def _sample_group(_prompt_text):
_messages = [{"role": "user", "content": _prompt_text}]
_model_input = renderer.build_generation_prompt(_messages)
_result = await sampling_client.sample(
prompt=_model_input, num_samples=_GROUP_SIZE, sampling_params=params
)
return _prompt_text, _result
_results = await asyncio.gather(*[_sample_group(p) for p in prompts])
total_completions = 0
for _prompt_text, _result in _results:
completions = []
for _seq in _result.sequences:
# Collect all results
_response_msg, _ = renderer.parse_response(_seq.tokens)
completions.append(get_text_content(_response_msg))
total_completions += len(completions)
print(f"Q: {_prompt_text}")
print(f" ({len(completions)} completions, showing first): {completions[0][:100]}...\n")
batch_time = time.time() - _start
print(f"Total: {total_completions} completions in {batch_time:.1f}s")
print(f"Throughput: {total_completions / batch_time:.1f} completions/second")
Output
Q: What causes thunder?
(4 completions, showing first): Here's a thinking process that leads to the explanation of thunder:
1. **Analyze the Request:**
...
Q: Write a haiku about the ocean.
(4 completions, showing first): Thinking Process:
1. **Analyze the Request:**
* Topic: The ocean.
* Form: Haiku (5-7-5...
Q: What is the capital of New Zealand?
(4 completions, showing first): Thinking Process:
1. **Identify the core question:** The user is asking for the capital city of Ne...
Q: Explain what a hash table is in two sentences.
(4 completions, showing first): Thinking Process:
1. **Analyze the Request:**
* Topic: Hash Table.
* Constraint: Exact...
Q: Name three inventions from the 19th century.
(4 completions, showing first): Thinking Process:
1. **Analyze the Request:**
* Task: Name three inventions.
* Constra...
Q: Why do leaves change color in autumn?
(4 completions, showing first): Here's a thinking process that leads to the explanation of why leaves change color in autumn:
1. *...
Q: Translate to Spanish: The library closes at nine.
(4 completions, showing first): We need to translate the sentence "The library closes at nine." from English to Spanish. First, let'...
Q: What is the smallest prime number greater than 50?
(4 completions, showing first): Thinking Process:
1. **Analyze the Request:** The user is asking for the smallest prime number tha...
Total: 32 completions in 2.2s
Throughput: 14.8 completions/second
Summary
SDK requests share one APIFuture interface: call a method to submit work, then await the future for its final value (or use .result() in synchronous code). For efficient sampling, submit concurrent requests before collecting them with asyncio.gather, and use num_samples to generate multiple completions per call.
Next steps:
- Tutorial 104: First RL: Uses this exact pattern -- sample many completions, grade them with a reward function, and train with GRPO.
- Concurrency and Async Operations: Core rules for futures, synchronous waiting, and migrating SDK calls.