> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Reinforcement learning with GRPO quickstart

> Train a model to solve grade-school math with GRPO, monitor the run, and retrieve the trained model.

[Group relative policy optimization (GRPO)](https://docs.together.ai/docs/post-training/grpo.md) is a reinforcement learning (RL) method that trains a model from a reward signal instead of example responses. The model generates several answers to each prompt, a reward function scores them, and the update favors the answers that beat their group's average.

In this tutorial, you run a ready-made recipe from the [post-training cookbook](https://docs.together.ai/docs/post-training/cookbook-overview.md) that trains a low-rank adapter (LoRA) for Qwen3.6 35B A3B on the [GSM8K](https://huggingface.co/datasets/openai/gsm8k) grade-school math dataset. An exact-match reward checks each final answer against the reference. By the end, you have a trained adapter registered in your project and know what each part of the training loop does.

The recipe runs 200 steps and takes about 4 hours. The model resource bills [per GPU-minute](https://docs.together.ai/docs/post-training/pricing.md) from the moment it's ready until it stops, at about \$96 per hour on the model's default GPUs. The full run costs about \$400, so make sure your available credit covers it.

> [!TIP]
>
> For more control over the training loop, [write it yourself](https://docs.together.ai/docs/post-training/training-loop.md) with the Python SDK.

## Requirements

Before you begin, make sure you have:

* Python 3.11 or later and [uv](https://docs.astral.sh/uv/getting-started/installation/).
* The [Together CLI](https://docs.together.ai/reference/cli/getting-started.md), version 3.0.0 or later, to download the trained model. Install or upgrade it with `uv tool install "together[cli]" --upgrade`.
* A [Together AI account](https://api.together.ai/) with enough credit to run your model resource for four hours, the [minimum for creating one](https://docs.together.ai/docs/post-training/pricing.md#minimum-credit-balance).
* Your API key exported as `TOGETHER_API_KEY`. Get one from [API keys](https://api.together.ai/settings/api-keys) in the Together AI console.
* A Weights & Biases API key exported as `WANDB_API_KEY`. Most cookbook recipes log to Weights & Biases, including the one in this quickstart. To run without an account, export `WANDB_MODE=offline` instead.
* Optionally, a Hugging Face token exported as `HF_TOKEN` for gated models or datasets.

```bash theme={null}
export TOGETHER_API_KEY=<your_key>
export WANDB_API_KEY=<your_wandb_key>     # or: export WANDB_MODE=offline
export HF_TOKEN=<your_huggingface_token>  # gated models and datasets only
```

## Step 1: Clone the cookbook

Clone the cookbook and install its dependencies, including the Together Python SDK:

```bash theme={null}
git clone https://github.com/togethercomputer/post-training-cookbook.git
cd post-training-cookbook
uv sync --locked
```

## Step 2: Review the recipe

Open the GSM8K recipe, `recipes/grpo/gsm8k.py`. It passes a config to `train_single_turn`, the cookbook's single-turn GRPO training loop:

```python recipes/grpo/gsm8k.py theme={null}
from post_training_cookbook.grpo.config import GRPOConfig
from post_training_cookbook.grpo.train import train_single_turn

config = GRPOConfig(
    model="Qwen/Qwen3.6-35B-A3B",
    dataset="gsm8k",
    enable_thinking=False,
    num_examples=4000,
    num_val_examples=200,
    batch_size=16,
    group_size=8,
    max_sampling_attempts=3,
    num_steps=200,
    learning_rate=1e-5,
    max_model_length=1536,
    max_prompt_length=768,
    max_sample_tokens=768,
    training_checkpoint_interval=20,
    inference_checkpoint_interval=20,
    validation_interval=100,
    wandb_project="grpo-gsm8k",
)

if __name__ == "__main__":
    train_single_turn(config)
```

The `dataset` field selects GSM8K from the cookbook's dataset registry, which supplies the prompt format and the reward. GSM8K reference answers end with `#### <number>`, and the prompt asks the model to end its solution the same way.

The reward function reads the number after `####` in each completion, or a `\boxed{}` answer when there's no `####`. It returns `1.0` if that number matches the reference and `0.0` otherwise. A completion with neither marker scores `0.0`, even when it states the right number.

These other settings shape the run:

| Setting | What it does |
| - | - |
| `model="Qwen/Qwen3.6-35B-A3B"` | A mixture-of-experts model with 3B active parameters, so it samples quickly for its size. |
| `batch_size=16`, `group_size=8` | 16 prompts and 8 completions per prompt, so 128 completions per step. Larger groups give steadier advantage estimates. |
| `max_sampling_attempts=3` | Turns on [dynamic sampling](https://docs.together.ai/docs/post-training/grpo-recipes.md#dynamic-sampling), which the default (`1`) leaves off. If fewer than 16 groups have varied rewards, the step draws another batch, up to three batches in total. If no batch has any group with varied rewards, the recipe skips the step and doesn't update the model. |
| `max_model_length=1536`, `max_prompt_length=768`, `max_sample_tokens=768` | GSM8K solutions are short. The recipe drops prompts over 768 tokens and caps completions at 768 tokens. The [overlong penalty](https://docs.together.ai/docs/post-training/grpo-recipes.md#overlong-responses), on by default, lowers the reward of completions longer than 672 tokens. |
| `enable_thinking=False` | Turns off the model's `<think>` output so completions stay inside the 768-token budget. |
| `training_checkpoint_interval=20`, `inference_checkpoint_interval=20` | Saves both checkpoint types every 20 steps. |
| `validation_interval=100` | Runs greedy validation every 100 steps and logs `val/accuracy`. |

Each step of `train_single_turn` samples `group_size` completions per prompt, scores them, and computes each completion's advantage relative to its group. It then runs `forward_backward` with the GRPO loss, `optim_step`, and `weights_sync`, so the next step samples from the updated model. See [How Together post-training runs GRPO](https://docs.together.ai/docs/post-training/grpo.md#how-together-post-training-runs-grpo) for details.

## Step 3: Run training

Run the recipe:

```bash theme={null}
uv run python recipes/grpo/gsm8k.py
```

The recipe provisions a model resource with one generator replica, starts a session on it, and waits for both to become ready. Then it prints a few lines per step. You should see output like this:

```text theme={null}
Dataset: gsm8k (openai/gsm8k)
Dataset size: 4000 train, 200 val
Training for 200 steps (1 epochs, 16 prompts x 8 samples per step)
[model-resources:9e4a1c7b-5d2f-4b8e-a3c6-7f0d2e9b1a54] created, waiting for READY status...
...
[model-resources:9e4a1c7b-5d2f-4b8e-a3c6-7f0d2e9b1a54] ready in 284.3s
[session:3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23] created, waiting for RUNNING status...
...
[session:3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23] ready in 63.8s
Session ready: 3f8b2c7d-91a4-4d2e-b6f0-5c1e9a7d4b23
Run artifacts: runs/Qwen3.6-35B-A3B_grpo-gsm8k_1.00e-05_16x8_20261001T153000Z
[3f8b2c7d] [step 0] VALIDATION accuracy=0.120 mean_len=312 elapsed=45.2s
[3f8b2c7d] [step 0] accuracy=0.188 reward=0.188 penalized=0.00% zero_var_skip=11.11% clipped=0.00% attempts=2
[3f8b2c7d] [step 1] loss=0.031200 lr=1.00e-06 elapsed=107.6s
[3f8b2c7d] [step 1] timing: collect=48.1s fb=9.6s optim=4.6s total=62.3s
[3f8b2c7d] [step 1] accuracy=0.203 reward=0.203 penalized=0.00% zero_var_skip=5.88% clipped=0.00% attempts=2
...
```

The bracketed prefix on each step line, `[3f8b2c7d]`, is the first eight characters of the session ID.

## Step 4: Monitor the run

The step log and the `grpo-gsm8k` project in Weights & Biases report the same metrics. On a first run, watch these:

| Step log | Weights & Biases | What to look for |
| - | - | - |
| `VALIDATION accuracy` | `val/accuracy` | The fraction of validation problems answered correctly with greedy sampling. This is the main progress signal, and it should rise and then plateau. |
| `accuracy` | `accuracy` | Training accuracy, counted only over groups that trained, so it isn't comparable to validation accuracy. It should trend upward. |
| `zero_var_skip` | `zero_variance_skip_ratio` | The fraction of groups skipped because every completion scored the same, leaving no learning signal. High values mean the task is too easy or too hard for the model right now. |

The `loss` line is a poor progress signal for GRPO, so track accuracy instead. [Monitor training runs](https://docs.together.ai/docs/post-training/monitoring.md) covers the other metrics and how to act on them.

## Step 5: Retrieve the trained model

The recipe registers each inference checkpoint in your project's model registry and prints each checkpoint's ID as it saves it. The run ends with the final pair:

```text theme={null}
[step 200] training checkpoint: <training_checkpoint_id>
[step 200] inference checkpoint: <inference_checkpoint_id>
```

To download the adapter with the CLI, you need its model registry ID. List the session's inference checkpoints to find it. The session ID is on the `Session ready` line of the training output and in the run's `manifest.json`:

```bash theme={null}
tg training ls-checkpoints --type inference --session <SESSION_ID>
```

The **Registry artifact** column holds each checkpoint's adapter registry ID and revision ID, as `ml_...@rv_...`. Download the final adapter's files with it:

```bash theme={null}
tg models download <MODEL_ID> ./gsm8k_adapter
```

The download contains only the adapter weights, in SGLang's multi-LoRA layout, which PEFT and Hugging Face Transformers can't load. To get the [base model with the adapter merged in](https://docs.together.ai/docs/post-training/checkpoints.md#download-the-weights), ready for any inference framework, pass the ID on the `[step 200] inference checkpoint` line to the Python SDK's `download_checkpoint` function.

To serve the model on Together AI instead, [deploy its inference checkpoint](https://docs.together.ai/docs/post-training/deployment.md).

> [!TIP]
>
> You trained a model with GRPO, watched it learn, and retrieved the result.

## Stop the model resource

The run stops the session and the model resource when it exits, so billing has already ended. To confirm, list the model resources you created most recently:

```bash theme={null}
tg training ls-resources --created-by me --limit 5
```

You should see this run's resource first, with the status `stopped`. If the run was interrupted, its resource can still be active. A `ready` resource is billing, and a `pending` or `creating` one starts billing once it's ready. Stop the resource with its ID from the list:

```python theme={null}
from together import Together

client = Together()
client.post_training.model_resources.stop("<MODEL_RESOURCES_ID>", force=True)
```

`force=True` also stops any session that's still attached. Without it, the API refuses to stop a resource while a session is active.

You can also stop the resource from the **Model resources** tab under [Post-Training](https://api.together.ai/post-training/resources) in the console. [Manage model resources](https://docs.together.ai/docs/post-training/model-resources.md) covers stopping, idle timers, and what a resource costs while it runs.

## Next steps

> [card] [Reinforcement learning with GRPO](https://docs.together.ai/docs/post-training/grpo.md)
>
> Learn how GRPO runs and how to design a reward.

> [card] [GRPO recipes](https://docs.together.ai/docs/post-training/grpo-recipes.md)
>
> Bring your own dataset and reward to the GSM8K recipe.

> [card] [Concepts](https://docs.together.ai/docs/post-training/concepts.md)
>
> Learn how work splits between your code and Together AI GPUs, and what model resources, sessions, and checkpoints do.

> [card] [Deploy a checkpoint](https://docs.together.ai/docs/post-training/deployment.md)
>
> Deploy the trained model on dedicated model inference.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.