How Together post-training runs DPO
DPO never samples from the model, so it runs on a trainer-only model resource (num_generator_replicas=0). The post-training service has no built-in DPO loss, so DPO trains with a custom loss function that your own process computes.
Before the first update, your process snapshots the reference policy by scoring every training and validation pair with forward and keeping each response’s log-probability. Each training step then runs:
forward: The trainer returns the current policy’s log-probability of each target token, without accumulating gradients.- Local loss: Your process computes the loss in PyTorch from the current and reference log-probabilities, then calls
.backward()to get the loss’s gradient with respect to each log-probability. custom_forward_backward: Your process sends those gradients to the trainer, which repeats the forward pass and backpropagates them through the model to accumulate parameter gradients.optim_step: The optimizer applies the accumulated gradients.
Monitor preference accuracy
DPO validation measures preference accuracy, the fraction of held-out pairs where the policy’s implicit reward favors the chosen response. In a recipe run, watch the validation metricsval/preference_accuracy and val/mean_margin rise.
Accuracy at step zero is 0. The policy still equals the reference, so every pair ties, and ties count as wrong.
Next steps
DPO recipes
Run DPO from the cookbook recipe on HelpSteer3 or your own preference pairs.
Write a custom loss function
Snapshot the reference policy and compute a PyTorch loss in your own loop with the Python SDK.
Supervised fine-tuning
Warm up a model on demonstrations before preference training.
Loss functions
See what a custom loss function can compute, and how it compares with the built-in ones.