Skip to main content
Group relative policy optimization (GRPO) trains a model from rewards you compute for groups of generated completions. Updates favor completions above their group’s mean reward and suppress those below it. Use it when you can score outputs but can’t write ideal responses, or to improve accuracy beyond your demonstrations.

How Together post-training runs GRPO

GRPO cycle: generators sample completions, your code computes rewards and advantages, and the trainer computes gradients, updates weights, and synchronizes generators. Low-rank adapter (LoRA) training freezes the base weights. Supervised fine-tuning and direct preference optimization skip generation, rewards, and generator synchronization.
Each step has six parts:
  1. sample: Generators return group_size completions per prompt, along with each generated token’s log-probability.
  2. Reward: Your reward function scores each completion.
  3. Advantages: Your code subtracts the group’s mean reward from each completion’s reward.
  4. forward_backward: With LOSS_TYPE_GRPO, the trainer computes the clipped policy-gradient loss and accumulates gradients. Each sample carries completion tokens, advantages, and the generator’s log-probabilities.
  5. optim_step: The optimizer applies the gradients with the learning rate, gradient clipping, and other optimizer settings in adam_params.
  6. weights_sync: The trainer publishes the updated weights to the generators before the next sample, unless an async training mode defers the sync.
Rewards and advantages run in your process, along with data loading and prompt construction, so they can be any code you write. The other four parts are operations that run on the model resource.

Rewards and advantages

Four completions score 1, 0, 1, and 0. Subtracting their mean of 0.5 gives advantages of +0.5, -0.5, +0.5, and -0.5. Groups with identical rewards are skipped, although generation is billed.
The cookbook computes advantage = reward - group_mean. It leaves advantages on the reward’s scale instead of dividing by the group’s standard deviation as in the GRPO paper. When every completion in a group gets the same reward, every advantage is zero, so the group carries no signal. Skip these zero-variance groups. Their generation is still billed. Group size: Larger groups, such as 16, give steadier advantages and fewer zero-variance groups, at the cost of proportionally more sampling. Temperature: When many groups score identically, raise the sampling temperature toward 1.0 for more varied completions. To exploit the current policy, lower it toward 0.5.

Write a reward function

The reward function is GRPO’s only learning signal. It scores one completion, usually against a target from the dataset row, and returns a float:
A good reward function:
  • Returns values in [0, 1]: Advantages stay on the reward’s scale, so a fixed range keeps them predictable.
  • Varies across completions: GRPO’s signal comes from the differences between completions in a group. Before training, score a sample of the base model’s outputs and check that the scores vary.
  • Is deterministic: The same prediction and target always score the same.
  • Isolates code execution: A reward that executes model output needs a timeout. Run it in isolation from your machine, for example in a container or a code sandbox.
  • Works off the main thread: A reward can run in a worker thread, for example when a sweep trains each adapter in its own thread. Timeouts built on signal.alarm fail there, so use a subprocess timeout instead, like the one in the code_output_reward example.
  • Is fast: It runs at least batch_size * group_size times per step.

Reward shapes

Binary: 1.0 for correct, 0.0 for wrong, like exact_match_reward. Use it for tasks with a single checkable answer, such as math and factual questions. Tiered: Partial credit at several levels. Use it when progress short of the right answer deserves credit, such as in multi-step problems or in puzzles where getting the format right is a large part of the task.
extract_answer and is_close stand in for your task’s own answer parsing and tolerance check. Continuous: A distance or similarity score. Use it for tasks where approximate answers have value, such as translation or summarization.
A cookbook recipe can also train on one reward but report a different metric.

Design guidance

  • Reward the format you want: If you want \boxed{} answers, give partial credit for the format even when the content is wrong. The model has to learn to emit the format before it can get the content right.
  • Penalize degenerate outputs: Return 0.0 for empty responses, repetition, or outputs that don’t attempt the task.
  • Guard against reward hacking: Watch val/accuracy alongside training reward. If training reward rises while validation accuracy falls, or val/mean_response_length keeps growing, the model has found a way to score well on your reward without solving the task.

Troubleshooting

  • Most groups score identically: If most completions are right, use harder data or raise the temperature. If most are wrong, use easier data or a stronger model. Raising the group size to 16 also leaves fewer zero-variance groups, at the cost of extra sampling.
  • Checkpoint creation fails with You need to complete at least one training step before you can create an inference checkpoint: Every step was skipped, although sampling was billed. If groups had identical rewards, adjust task difficulty.
  • fb/loss/ratio/mean is far from 1.0 on the first step: The trainer and generator initially share weights, so the ratio should start near 1.0. In a hand-built loop, where the key is loss/ratio/mean, check that you dropped the first value of each per-token array, so logprobs aligns with target_tokens. Misaligned advantages don’t affect this metric.
  • Out of memory or timeouts: Lower the batch size, the group size, or the maximum sequence length. For timeouts, you can also raise each call’s timeout.
  • The session isn’t active: It was stopped, hit an error, or expired when its resource reached its idle timeout (one hour by default). See Lifecycle, errors, and limits.

Next steps

GRPO recipes

Run GRPO from a cookbook recipe on a built-in or registered dataset.

Write a training loop

Sample, score, and train with the Python SDK.

Multi-turn environments

Train tool-use agents over whole episodes.

Loss functions

Choose a policy loss, set clipping, or add a Kullback-Leibler (KL) penalty.