Skip to main content
Group relative policy optimization (GRPO) is a reinforcement learning (RL) method that trains a model from a reward signal instead of example responses. The model generates several answers to each prompt, a reward function scores them, and the update favors the answers that beat their group’s average. In this tutorial, you run a ready-made recipe from the post-training cookbook that trains a low-rank adapter (LoRA) for Qwen3.6 35B A3B on the GSM8K grade-school math dataset. An exact-match reward checks each final answer against the reference. By the end, you have a trained adapter registered in your project and know what each part of the training loop does. The recipe runs 200 steps and takes about 4 hours. The model resource bills per GPU-minute from the moment it’s ready until it stops, at about $96 per hour on the model’s default GPUs. The full run costs about $400, so make sure your available credit covers it.
For more control over the training loop, write it yourself with the Python SDK.

Requirements

Before you begin, make sure you have:
  • Python 3.11 or later and uv.
  • The Together CLI, version 3.0.0 or later, to download the trained model. Install or upgrade it with uv tool install "together[cli]" --upgrade.
  • A Together AI account with enough credit to run your model resource for four hours, the minimum for creating one.
  • Your API key exported as TOGETHER_API_KEY. Get one from API keys in the Together AI console.
  • A Weights & Biases API key exported as WANDB_API_KEY. Most cookbook recipes log to Weights & Biases, including the one in this quickstart. To run without an account, export WANDB_MODE=offline instead.
  • Optionally, a Hugging Face token exported as HF_TOKEN for gated models or datasets.

Step 1: Clone the cookbook

Clone the cookbook and install its dependencies, including the Together Python SDK:

Step 2: Review the recipe

Open the GSM8K recipe, recipes/grpo/gsm8k.py. It passes a config to train_single_turn, the cookbook’s single-turn GRPO training loop:
recipes/grpo/gsm8k.py
The dataset field selects GSM8K from the cookbook’s dataset registry, which supplies the prompt format and the reward. GSM8K reference answers end with #### <number>, and the prompt asks the model to end its solution the same way. The reward function reads the number after #### in each completion, or a \boxed{} answer when there’s no ####. It returns 1.0 if that number matches the reference and 0.0 otherwise. A completion with neither marker scores 0.0, even when it states the right number. These other settings shape the run: Each step of train_single_turn samples group_size completions per prompt, scores them, and computes each completion’s advantage relative to its group. It then runs forward_backward with the GRPO loss, optim_step, and weights_sync, so the next step samples from the updated model. See How Together post-training runs GRPO for details.

Step 3: Run training

Run the recipe:
The recipe provisions a model resource with one generator replica, starts a session on it, and waits for both to become ready. Then it prints a few lines per step. You should see output like this:
The bracketed prefix on each step line, [3f8b2c7d], is the first eight characters of the session ID.

Step 4: Monitor the run

The step log and the grpo-gsm8k project in Weights & Biases report the same metrics. On a first run, watch these: The loss line is a poor progress signal for GRPO, so track accuracy instead. Monitor training runs covers the other metrics and how to act on them.

Step 5: Retrieve the trained model

The recipe registers each inference checkpoint in your project’s model registry and prints each checkpoint’s ID as it saves it. The run ends with the final pair:
To download the adapter with the CLI, you need its model registry ID. List the session’s inference checkpoints to find it. The session ID is on the Session ready line of the training output and in the run’s manifest.json:
The Registry artifact column holds each checkpoint’s adapter registry ID and revision ID, as ml_...@rv_.... Download the final adapter’s files with it:
The download contains only the adapter weights, in SGLang’s multi-LoRA layout, which PEFT and Hugging Face Transformers can’t load. To get the base model with the adapter merged in, ready for any inference framework, pass the ID on the [step 200] inference checkpoint line to the Python SDK’s download_checkpoint function. To serve the model on Together AI instead, deploy its inference checkpoint.
You trained a model with GRPO, watched it learn, and retrieved the result.

Stop the model resource

The run stops the session and the model resource when it exits, so billing has already ended. To confirm, list the model resources you created most recently:
You should see this run’s resource first, with the status stopped. If the run was interrupted, its resource can still be active. A ready resource is billing, and a pending or creating one starts billing once it’s ready. Stop the resource with its ID from the list:
force=True also stops any session that’s still attached. Without it, the API refuses to stop a resource while a session is active. You can also stop the resource from the Model resources tab under Post-Training in the console. Manage model resources covers stopping, idle timers, and what a resource costs while it runs.

Next steps

Reinforcement learning with GRPO

Learn how GRPO runs and how to design a reward.

GRPO recipes

Bring your own dataset and reward to the GSM8K recipe.

Concepts

Learn how work splits between your code and Together AI GPUs, and what model resources, sessions, and checkpoints do.

Deploy a checkpoint

Deploy the trained model on dedicated model inference.