Skip to content
View as Markdown

Forecasting with RL

Fine-tune Qwen3.8-27B or GLM-5.3 with Tinker on binary markets from the Prophet Arena subset using a chronological split and Brier reward.

Run

export TINKER_API_KEY=...

# Optionally download, verify, and inspect the data split
python -m tinker_cookbook.recipes.forecasting.data

# Train the forecasting model
python -m tinker_cookbook.recipes.forecasting.train

# Or train with GLM-5.3 instead
python -m tinker_cookbook.recipes.forecasting.train \
    model_name=zai-org/GLM-5.3:peft:262144 \
    renderer_name=glm5_3_high_reasoning

Prompt

Task: Given a binary prediction-market question and information available at a historical snapshot, forecast the probability that the market will resolve YES. The model does not use tools or live search in this recipe.

Here is the exact prompt template we use:

Forecast whether this market will resolve YES using information available through {snapshot time}.

Event:
{event title}

Market:
{market}

Reference material:
{reference material available at the snapshot}

Resolution criteria:
{resolution criteria}

Market close time: {close time}

Output only the probability of YES as a number between 0 and 1.

An example from the pinned dataset is below. Note that this event bundles the markets PSG, Real Madrid, and Tie, so it becomes three questions. Below is the Real Madrid one, which resolved NO.

Forecast whether this market will resolve YES using information available through 2025-07-08T12:01:09.330143+00:00.

Event:
PSG vs Real Madrid

Market:
Real Madrid

Reference material:
1. 2024–25 Paris Saint-Germain FC season
This article details PSG's 2024–25 season, highlighting their achievements including winning Ligue 1, the Coupe de France, and the UEFA Champions League, marking their first continental treble. It emphasizes the team's strong performance under manager Luis Enrique and the significant contributions of key players like Ousmane Dembélé, who was the season's top scorer with 33 goals. This information is relevant as it showcases PSG's recent form and success leading up to their match against Real Madrid.

2. 2025 FIFA Club World Cup Group H
This article outlines the structure and participants of Group H in the 2025 FIFA Club World Cup, which includes Real Madrid. It provides insights into Real Madrid's recent international competition schedule and performance. Understanding their participation and results in this tournament can offer context for their preparedness and form ahead of the match against PSG.

3. Ousmane Dembélé
This article provides an overview of Ousmane Dembélé's career, focusing on his impactful tenure at PSG since joining in August 2023. It highlights his contributions, including scoring 33 goals in the 2024–25 season and playing a pivotal role in PSG's treble-winning campaign. Dembélé's form and performance are crucial factors to consider when predicting the outcome of the upcoming match against Real Madrid.

...three more sources...

Resolution criteria:
The market resolves YES if this condition occurs: Real Madrid.

Market close time: 2025-07-09T21:06:41.016512+00:00

Output only the probability of YES as a number between 0 and 1.

Prompts use the event, named market, reference material, and timestamp from its earliest snapshot, together with the market close time. Market prices and outcome fields are omitted.

Data

One CSV of 1,200 snapshots covering 869 events.

An event bundles one or more markets (median of 2 markets but occasionally over 100), and each market becomes its own binary question, with resolution criteria derived from the market name. The 1,200 rows expand to 5,232 questions.

Each row is one snapshot, and the loader keeps the earliest snapshot for each market. Note that the question never changes between snapshots, only the attached news and the market price, so the earliest snapshot is the longest-horizon and least informed version of the question.

Events that closed before October 20, 2025 form the training set. Events that were first observed on that cutoff date or later form the validation set, with every market of an event kept on the same side. Events that started before the cutoff and had not closed by it are excluded. That splits 869 events into 613 training and 207 validation, dropping 49 at the boundary. This produces 3,587 and 1,180 questions, which the default caps trim to 1,024 and 256.

The recipe downloads a fixed published version and verifies the CSV contents, so later dataset updates do not change a rerun.

Temporal split methodology

This cookbook specifically uses a cutoff of October 20, 2025 UTC, but feel free to adjust based on the model you are using.

  • Knowledge cutoff: Try to find the knowledge cutoff for the model you are post-training. Ideally its pretraining cutoff predates the forecasting questions. Qwen3.8 has been reported to self-identify an early-2025 cutoff, although this is not publisher-verified. Adjust the model or split date to reduce outcome contamination.
  • Training split: Events whose markets all closed before the cutoff.
  • Validation split: Events first observed on or after the cutoff.

We further refine our data via:

  • Boundary events: Events open across the cutoff are excluded.
  • Deduplication: Each event-market pair becomes one question using its earliest snapshot, and all markets from the same event remain in the same split.

Configuration

Model

Either:

These reasoning effort settings were the best we found in our sweeps. Note that Qwen at low effort generates about 600 tokens per forecast, which is actually significantly more than GLM at high effort which generates about 100.

Both use LoRA rank 32, a 1e-4 learning rate, and 32 training forecasts per question. When switching to another model, use that model's recommended renderer and rerun the relevant hyperparameter ablations.

Training loop

  • 16 questions per step
  • 128 training steps (two passes over the 1,024 training questions)

One epoch is 64 steps. Most of the improvement comes with the first epoch, so setting max_steps=64 is a reasonable way to shorten the training time.

Validation

  • every 16 steps and after the final update
  • 8 forecasts per question

Reward

For probability p and outcome y:

Brier reward = 1 - (p - y)^2

This is a strictly proper scoring rule; malformed answers receive 0. See Outcome-based Reinforcement Learning to Predict the Future for the outcome-based forecasting setup.

Result

With Qwen3.8-27B, we produced the following for reference:

Step Validation Brier reward Validation accuracy Validation AUC Valid format
0 0.7998 72.92% 0.7959 99.80%
16 0.8166 74.32% 0.7981 100.00%
32 0.8207 73.88% 0.8025 100.00%
48 0.8222 73.97% 0.8045 100.00%
64 0.8262 74.54% 0.8102 100.00%
80 0.8202 73.61% 0.8026 100.00%
96 0.8071 72.22% 0.8050 99.90%
112 0.8288 74.78% 0.8145 100.00%
128 0.8294 75.34% 0.8175 100.00%

With GLM-5.3:

Step Validation Brier reward Validation accuracy Validation AUC Valid format
0 0.7952 72.53% 0.8088 98.63%
16 0.8254 74.78% 0.8122 99.95%
32 0.8224 74.41% 0.8131 99.95%
48 0.8058 72.36% 0.7808 100.00%
64 0.8413 77.32% 0.8387 100.00%
80 0.8373 76.32% 0.8289 100.00%
96 0.8309 75.32% 0.8265 100.00%
112 0.8387 77.05% 0.8331 100.00%
128 0.8475 78.30% 0.8495 100.00%

This shows consistent improvement on temporally held-out validation questions and suggests that the learned forecasting behavior generalizes beyond the training events.

For scale, always answering 0.5 scores a Brier reward of 0.7500 on this validation split, and always answering the training base rate scores 0.7709.

AUC measures how well the forecasts rank YES outcomes above NO outcomes, independent of calibration. It is computed over the whole validation set at each evaluation and logged as test/env/all/auc. A model with no discrimination ability scores 0.5, whereas a model with perfect discrimination scores 1.0.

License

Prophet Arena data is distributed under the MIT license; see the dataset card for terms.