> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Reinforcement learning with GRPO

> Train a model with group relative policy optimization, and write a reward function the model can learn from.

Group relative policy optimization (GRPO) trains a model from rewards you compute for groups of generated completions. Updates favor completions above their group's mean reward and suppress those below it. Use it when you can score outputs but can't write ideal responses, or to improve accuracy beyond your demonstrations.

## How Together post-training runs GRPO

![GRPO cycle: generators sample completions, your code computes rewards and advantages, and the trainer computes gradients, updates weights, and synchronizes generators. Low-rank adapter (LoRA) training freezes the base weights. Supervised fine-tuning and direct preference optimization skip generation, rewards, and generator synchronization.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/grpo-training-loop.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=a73cd0cb26528aa664c90e5be06751ee)

Each step has six parts:

1. **`sample`:** Generators return `group_size` completions per prompt, along with each generated token's log-probability.
2. **Reward:** Your reward function scores each completion.
3. **Advantages:** Your code subtracts the group's mean reward from each completion's reward.
4. **`forward_backward`:** With `LOSS_TYPE_GRPO`, the trainer computes the [clipped policy-gradient loss](https://docs.together.ai/docs/post-training/loss-functions.md#clipping) and accumulates gradients. Each sample carries completion tokens, advantages, and the generator's log-probabilities.
5. **`optim_step`:** The optimizer applies the gradients with the learning rate, gradient clipping, and other [optimizer settings](https://docs.together.ai/reference/post-training/operations-optim-step-create.md) in `adam_params`.
6. **`weights_sync`:** The trainer publishes the updated weights to the generators before the next `sample`, unless an [async training](https://docs.together.ai/docs/post-training/multi-turn-recipes.md#async-training) mode defers the sync.

Rewards and advantages run in your process, along with data loading and prompt construction, so they can be any code you write. The other four parts are operations that run on the model resource.

## Rewards and advantages

![Four completions score 1, 0, 1, and 0. Subtracting their mean of 0.5 gives advantages of +0.5, -0.5, +0.5, and -0.5. Groups with identical rewards are skipped, although generation is billed.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/group-advantages.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=afe238275df9a8f2f8700cb05ad33307)

The cookbook computes `advantage = reward - group_mean`. It leaves advantages on the reward's scale instead of dividing by the group's standard deviation as in the GRPO paper.

When every completion in a group gets the same reward, every advantage is zero, so the group carries no signal. Skip these *zero-variance groups*. Their generation is still billed.

**Group size:** Larger groups, such as `16`, give steadier advantages and fewer zero-variance groups, at the cost of proportionally more sampling.

**Temperature:** When many groups score identically, raise the sampling temperature toward `1.0` for more varied completions. To exploit the current policy, lower it toward `0.5`.

## Write a reward function

The reward function is GRPO's only learning signal. It scores one completion, usually against a target from the dataset row, and returns a float:

```python theme={null}
def exact_match_reward(prediction: str, target: str) -> float:
    return 1.0 if prediction.strip() == target.strip() else 0.0
```

A good reward function:

* **Returns values in `[0, 1]`:** Advantages stay on the reward's scale, so a fixed range keeps them predictable.
* **Varies across completions:** GRPO's signal comes from the differences between completions in a group. Before training, [score a sample of the base model's outputs](https://docs.together.ai/docs/post-training/evaluate.md#check-a-baseline-before-training) and check that the scores vary.
* **Is deterministic:** The same prediction and target always score the same.
* **Isolates code execution:** A reward that executes model output needs a timeout. Run it in isolation from your machine, for example in a container or a [code sandbox](https://docs.together.ai/docs/together-code-sandbox.md).
* **Works off the main thread:** A reward can run in a worker thread, for example when a [sweep](https://docs.together.ai/docs/post-training/multi-lora.md#run-a-sweep) trains each adapter in its own thread. Timeouts built on `signal.alarm` fail there, so use a subprocess timeout instead, like the one in the [`code_output_reward` example](https://docs.together.ai/docs/post-training/grpo-recipes#register-a-dataset-and-reward).
* **Is fast:** It runs at least `batch_size * group_size` times per step.

### Reward shapes

**Binary:** `1.0` for correct, `0.0` for wrong, like `exact_match_reward`. Use it for tasks with a single checkable answer, such as math and factual questions.

**Tiered:** Partial credit at several levels. Use it when progress short of the right answer deserves credit, such as in multi-step problems or in puzzles where getting the format right is a large part of the task.

```python theme={null}
def tiered_reward(prediction: str, target: str) -> float:
    score = 0.0
    answer = extract_answer(prediction)
    if answer is not None:
        score += 0.2  # followed the format
    if answer and is_close(answer, target):
        score += 0.4  # close to the answer
    if answer == target:
        score += 0.4  # exact match
    return score
```

`extract_answer` and `is_close` stand in for your task's own answer parsing and tolerance check.

**Continuous:** A distance or similarity score. Use it for tasks where approximate answers have value, such as translation or summarization.

```python theme={null}
from difflib import SequenceMatcher


def similarity_reward(prediction: str, target: str) -> float:
    return SequenceMatcher(None, prediction.strip(), target.strip()).ratio()
```

A cookbook recipe can also [train on one reward but report a different metric](https://docs.together.ai/docs/post-training/grpo-recipes.md#report-a-separate-metric).

### Design guidance

* **Reward the format you want:** If you want `\boxed{}` answers, give partial credit for the format even when the content is wrong. The model has to learn to emit the format before it can get the content right.
* **Penalize degenerate outputs:** Return `0.0` for empty responses, repetition, or outputs that don't attempt the task.
* **Guard against reward hacking:** Watch [`val/accuracy`](https://docs.together.ai/docs/post-training/monitoring#validation-metrics) alongside training reward. If training reward rises while validation accuracy falls, or `val/mean_response_length` keeps growing, the model has found a way to score well on your reward without solving the task.

## Troubleshooting

* **Most groups score identically:** If most completions are right, use harder data or raise the temperature. If most are wrong, use easier data or a stronger model. Raising the group size to `16` also leaves fewer zero-variance groups, at the cost of extra sampling.
* **Checkpoint creation fails with `You need to complete at least one training step before you can create an inference checkpoint`:** Every step was skipped, although sampling was billed. If groups had identical rewards, adjust task difficulty.
* **`fb/loss/ratio/mean` is far from `1.0` on the first step:** The trainer and generator initially share weights, so the ratio should start near `1.0`. In a hand-built loop, where the key is `loss/ratio/mean`, check that you [dropped the first value of each per-token array](https://docs.together.ai/docs/post-training/samples.md#build-samples-by-hand), so `logprobs` aligns with `target_tokens`. Misaligned `advantages` don't affect this metric.
* **Out of memory or timeouts:** Lower the batch size, the group size, or the maximum sequence length. For timeouts, you can also raise each call's `timeout`.
* **The session isn't active:** It was stopped, hit an error, or expired when its resource reached its idle timeout (one hour by default). See [Lifecycle, errors, and limits](https://docs.together.ai/docs/post-training/limits-and-errors.md#troubleshooting).

## Next steps

> [card] [GRPO recipes](https://docs.together.ai/docs/post-training/grpo-recipes.md)
>
> Run GRPO from a cookbook recipe on a built-in or registered dataset.

> [card] [Write a training loop](https://docs.together.ai/docs/post-training/training-loop.md)
>
> Sample, score, and train with the Python SDK.

> [card] [Multi-turn environments](https://docs.together.ai/docs/post-training/multi-turn-environments.md)
>
> Train tool-use agents over whole episodes.

> [card] [Loss functions](https://docs.together.ai/docs/post-training/loss-functions.md)
>
> Choose a policy loss, set clipping, or add a Kullback-Leibler (KL) penalty.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.