> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Direct preference optimization

> Train a model on preference pairs with direct preference optimization, run as a custom loss function.

Direct preference optimization (DPO) trains a model on *preference pairs*, two responses to the same prompt where one is chosen over the other. Updates raise the likelihood of the chosen response relative to the rejected one. DPO measures each likelihood against the *reference policy*, a frozen snapshot of the model from before training. Use it when you have preference data, such as human ratings, reward-model scores, or comparisons between systems.

[Fine-tuning jobs](https://docs.together.ai/docs/fine-tuning/preference-tuning.md) also run DPO, with a fixed data format and no code to write. To choose between them, see [Post-training vs. standard fine-tuning](https://docs.together.ai/docs/post-training/overview.md#post-training-vs-standard-fine-tuning).

## How Together post-training runs DPO

DPO never samples from the model, so it runs on a trainer-only model resource (`num_generator_replicas=0`). The post-training service has no built-in DPO loss, so DPO trains with a [custom loss function](https://docs.together.ai/docs/post-training/loss-functions.md#how-a-custom-loss-function-runs) that your own process computes.

Before the first update, your process snapshots the reference policy by scoring every training and validation pair with `forward` and keeping each response's log-probability. Each training step then runs:

1. **`forward`:** The trainer returns the current policy's log-probability of each target token, without accumulating gradients.
2. **Local loss:** Your process computes the loss in PyTorch from the current and reference log-probabilities, then calls `.backward()` to get the loss's gradient with respect to each log-probability.
3. **`custom_forward_backward`:** Your process sends those gradients to the trainer, which repeats the forward pass and backpropagates them through the model to accumulate parameter gradients.
4. **`optim_step`:** The optimizer applies the accumulated gradients.

## Monitor preference accuracy

DPO validation measures *preference accuracy*, the fraction of held-out pairs where the policy's implicit reward favors the chosen response. In a recipe run, watch the [validation metrics](https://docs.together.ai/docs/post-training/monitoring.md#dpo-metrics) `val/preference_accuracy` and `val/mean_margin` rise.

Accuracy at step zero is `0`. The policy still equals the reference, so every pair ties, and ties count as wrong.

## Next steps

> [card] [DPO recipes](https://docs.together.ai/docs/post-training/dpo-recipes.md)
>
> Run DPO from the cookbook recipe on HelpSteer3 or your own preference pairs.

> [card] [Write a custom loss function](https://docs.together.ai/docs/post-training/custom-loss-functions.md)
>
> Snapshot the reference policy and compute a PyTorch loss in your own loop with the Python SDK.

> [card] [Supervised fine-tuning](https://docs.together.ai/docs/post-training/sft.md)
>
> Warm up a model on demonstrations before preference training.

> [card] [Loss functions](https://docs.together.ai/docs/post-training/loss-functions.md)
>
> See what a custom loss function can compute, and how it compares with the built-in ones.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.