> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-turn GRPO

> Multi-turn GRPO trains tool-use agents on whole episodes, where an environment responds to each action and scores the outcome.

Multi-turn group relative policy optimization (GRPO) trains a model on whole episodes rather than on one response per prompt. In each episode, the model takes an action (an assistant message that may carry tool calls), an environment responds, and the loop repeats until the episode ends.

The environment scores each episode when it ends, and every assistant turn in the episode trains on that reward. The [reward design guidance](https://docs.together.ai/docs/post-training/grpo.md#design-guidance) for single-turn GRPO applies to episode rewards too.

Use multi-turn GRPO for tool-use agents, multi-step reasoning against an external system, and any task where the model has to act more than once to earn a reward. For tasks the model completes in one response, use [single-turn GRPO](https://docs.together.ai/docs/post-training/grpo.md).

## Requirements

Multi-turn GRPO needs a [model with generator support](https://docs.together.ai/docs/post-training/supported-models.md#catalog) and a model resource with `num_generator_replicas` of one or more.

## How an episode runs

Each episode starts from a *task*, which carries the prompt, the target the reward checks against, and any metadata the environment needs. The loop then alternates between the model and the environment:

1. The environment returns the task's opening messages, typically a system prompt and the user prompt.
2. The loop renders the messages and the environment's tool specs into a prompt with the model's chat template, samples a reply from the model, then parses it into an assistant message with any tool calls.
3. The environment executes any tool calls in the assistant message and appends the results to the conversation history. It returns the updated history, a reward, and whether the episode is over.
4. Steps 2 and 3 repeat until the environment marks the episode complete or the episode is [truncated](#truncation).

The loop runs a group of episodes for each task and computes advantages across the group, as single-turn GRPO does with the group of completions it samples for each prompt.

With the cookbook, you write the environment as a [subclass of `Env`](https://docs.together.ai/docs/post-training/multi-turn-recipes#write-an-environment), and the cookbook runs the loop. When you [write your own loop](https://docs.together.ai/docs/post-training/training-loop.md), your code both runs the loop and acts as the environment.

## How episodes become training samples

![A multi-turn episode flattened into one token stream of user context, assistant action, tool context, and assistant action. Context segments get loss mask 0 and aren't trained on. Action segments get loss mask 1, are trained on, and share the episode's advantage.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/multi-turn-loss-masks.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=3c66fdd8684ac5a2e154ab60883e8eaa)

An episode is a sequence of alternating turns, but the trainer sees flat token sequences. The loop flattens each episode into context segments, which hold the prompt and the tool results, and action segments, which hold the model's replies.

Context segments get loss mask `0`, so the model isn't trained to predict them. Action segments get loss mask `1`, and every action in the episode carries the episode's advantage.

Each action segment holds the tokens the generator sampled and the log-probabilities it returned for them, not a re-rendering of the parsed reply. The trainer's importance ratio compares its own log-probabilities with the generator's token by token, so both must cover the same tokens.

When the [chat template strips earlier content from history](https://docs.together.ai/docs/post-training/concepts.md#chat-templates-and-thinking), such as a reasoning model's `<think>` blocks, the loop starts a new training sequence at that point, so the tokens stay aligned with the log-probabilities the generator recorded.

## Truncation

An episode is *truncated* when it ends without the environment marking it complete. That happens when the conversation fills the context budget, the episode hits its turn limit, the environment cuts the episode short, or a sampled message can't be parsed. [Handle truncated episodes](https://docs.together.ai/docs/post-training/multi-turn-recipes.md#handle-truncated-episodes) covers how the cookbook penalizes and masks truncated episodes.

## Next steps

> [card] [Multi-turn recipes](https://docs.together.ai/docs/post-training/multi-turn-recipes.md)
>
> Write an environment and train on it with the cookbook.

> [card] [Reinforcement learning with GRPO](https://docs.together.ai/docs/post-training/grpo.md)
>
> Learn how a GRPO step runs and how rewards become advantages.

> [card] [Monitor training runs](https://docs.together.ai/docs/post-training/monitoring.md#multi-turn-metrics)
>
> Track how many turns episodes take and how they end.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.