How Together post-training runs GRPO

sample: Generators returngroup_sizecompletions per prompt, along with each generated token’s log-probability.- Reward: Your reward function scores each completion.
- Advantages: Your code subtracts the group’s mean reward from each completion’s reward.
forward_backward: WithLOSS_TYPE_GRPO, the trainer computes the clipped policy-gradient loss and accumulates gradients. Each sample carries completion tokens, advantages, and the generator’s log-probabilities.optim_step: The optimizer applies the gradients with the learning rate, gradient clipping, and other optimizer settings inadam_params.weights_sync: The trainer publishes the updated weights to the generators before the nextsample, unless an async training mode defers the sync.
Rewards and advantages

advantage = reward - group_mean. It leaves advantages on the reward’s scale instead of dividing by the group’s standard deviation as in the GRPO paper.
When every completion in a group gets the same reward, every advantage is zero, so the group carries no signal. Skip these zero-variance groups. Their generation is still billed.
Group size: Larger groups, such as 16, give steadier advantages and fewer zero-variance groups, at the cost of proportionally more sampling.
Temperature: When many groups score identically, raise the sampling temperature toward 1.0 for more varied completions. To exploit the current policy, lower it toward 0.5.
Write a reward function
The reward function is GRPO’s only learning signal. It scores one completion, usually against a target from the dataset row, and returns a float:- Returns values in
[0, 1]: Advantages stay on the reward’s scale, so a fixed range keeps them predictable. - Varies across completions: GRPO’s signal comes from the differences between completions in a group. Before training, score a sample of the base model’s outputs and check that the scores vary.
- Is deterministic: The same prediction and target always score the same.
- Isolates code execution: A reward that executes model output needs a timeout. Run it in isolation from your machine, for example in a container or a code sandbox.
- Works off the main thread: A reward can run in a worker thread, for example when a sweep trains each adapter in its own thread. Timeouts built on
signal.alarmfail there, so use a subprocess timeout instead, like the one in thecode_output_rewardexample. - Is fast: It runs at least
batch_size * group_sizetimes per step.
Reward shapes
Binary:1.0 for correct, 0.0 for wrong, like exact_match_reward. Use it for tasks with a single checkable answer, such as math and factual questions.
Tiered: Partial credit at several levels. Use it when progress short of the right answer deserves credit, such as in multi-step problems or in puzzles where getting the format right is a large part of the task.
extract_answer and is_close stand in for your task’s own answer parsing and tolerance check.
Continuous: A distance or similarity score. Use it for tasks where approximate answers have value, such as translation or summarization.
Design guidance
- Reward the format you want: If you want
\boxed{}answers, give partial credit for the format even when the content is wrong. The model has to learn to emit the format before it can get the content right. - Penalize degenerate outputs: Return
0.0for empty responses, repetition, or outputs that don’t attempt the task. - Guard against reward hacking: Watch
val/accuracyalongside training reward. If training reward rises while validation accuracy falls, orval/mean_response_lengthkeeps growing, the model has found a way to score well on your reward without solving the task.
Troubleshooting
- Most groups score identically: If most completions are right, use harder data or raise the temperature. If most are wrong, use easier data or a stronger model. Raising the group size to
16also leaves fewer zero-variance groups, at the cost of extra sampling. - Checkpoint creation fails with
You need to complete at least one training step before you can create an inference checkpoint: Every step was skipped, although sampling was billed. If groups had identical rewards, adjust task difficulty. fb/loss/ratio/meanis far from1.0on the first step: The trainer and generator initially share weights, so the ratio should start near1.0. In a hand-built loop, where the key isloss/ratio/mean, check that you dropped the first value of each per-token array, sologprobsaligns withtarget_tokens. Misalignedadvantagesdon’t affect this metric.- Out of memory or timeouts: Lower the batch size, the group size, or the maximum sequence length. For timeouts, you can also raise each call’s
timeout. - The session isn’t active: It was stopped, hit an error, or expired when its resource reached its idle timeout (one hour by default). See Lifecycle, errors, and limits.
Next steps
GRPO recipes
Run GRPO from a cookbook recipe on a built-in or registered dataset.
Write a training loop
Sample, score, and train with the Python SDK.
Multi-turn environments
Train tool-use agents over whole episodes.
Loss functions
Choose a policy loss, set clipping, or add a Kullback-Leibler (KL) penalty.