Skip to main content
Direct preference optimization (DPO) trains a model on preference pairs, two responses to the same prompt where one is chosen over the other. Updates raise the likelihood of the chosen response relative to the rejected one. DPO measures each likelihood against the reference policy, a frozen snapshot of the model from before training. Use it when you have preference data, such as human ratings, reward-model scores, or comparisons between systems. Fine-tuning jobs also run DPO, with a fixed data format and no code to write. To choose between them, see Post-training vs. standard fine-tuning.

How Together post-training runs DPO

DPO never samples from the model, so it runs on a trainer-only model resource (num_generator_replicas=0). The post-training service has no built-in DPO loss, so DPO trains with a custom loss function that your own process computes. Before the first update, your process snapshots the reference policy by scoring every training and validation pair with forward and keeping each response’s log-probability. Each training step then runs:
  1. forward: The trainer returns the current policy’s log-probability of each target token, without accumulating gradients.
  2. Local loss: Your process computes the loss in PyTorch from the current and reference log-probabilities, then calls .backward() to get the loss’s gradient with respect to each log-probability.
  3. custom_forward_backward: Your process sends those gradients to the trainer, which repeats the forward pass and backpropagates them through the model to accumulate parameter gradients.
  4. optim_step: The optimizer applies the accumulated gradients.

Monitor preference accuracy

DPO validation measures preference accuracy, the fraction of held-out pairs where the policy’s implicit reward favors the chosen response. In a recipe run, watch the validation metrics val/preference_accuracy and val/mean_margin rise. Accuracy at step zero is 0. The policy still equals the reference, so every pair ties, and ties count as wrong.

Next steps

DPO recipes

Run DPO from the cookbook recipe on HelpSteer3 or your own preference pairs.

Write a custom loss function

Snapshot the reference policy and compute a PyTorch loss in your own loop with the Python SDK.

Supervised fine-tuning

Warm up a model on demonstrations before preference training.

Loss functions

See what a custom loss function can compute, and how it compares with the built-in ones.