forward returns the model’s log-probabilities, your PyTorch code computes the loss and its gradients, and custom_forward_backward sends the gradients back to the trainer. How a custom loss function runs explains each step.
Read the log-probabilities with forward, not with forward_backward and return_loss_fn_outputs=True. Even with that flag, forward_backward runs a backward pass through its built-in loss function, so its gradients add to yours before optim_step applies them.
Run a custom loss step
Build each sample in yourbatch the same way as for supervised fine-tuning (SFT), with target_tokens and a weights row that marks the positions your loss function reads. The cross-entropy LossConfig on forward only selects which positions return a log-probability, so every sample needs both arrays, whatever your loss function computes.
Keep any other values your loss function needs, such as reference log-probabilities or advantages, in your own code, not in the sample. Capture them in the loss function, as the example does with ref.
This example implements regularized SFT, which adds a penalty to cross-entropy for moving the model’s log-probabilities away from a snapshot taken before training:
- Average over the trained positions only:
forwardreturns a real log-probability only where the sample’sweightsentry is non-zero, and0.0at every other position. The service ignores gradients at those zero-weight positions, but the zeros still count in your loss if you average over the whole row. Divide by the mask’s sum instead, as the example does withm.sum(). - Match each gradient list to its sample: Pass
custom_forward_backwardone gradient list per sample, in batch order. Each list must have one value per target position, the same length as the sample’starget_tokens. - Snapshot a frozen reference once: Call
forwardfor the reference before training starts. Recomputing the reference every step repeats the same forward passes. - Split a large batch across
forwardcalls:forwardscores the whole batch in one operation. If it runs out of memory or returns no log-probabilities, call it on smaller slices of the batch. If the call times out, use smaller slices or pass a longertimeout. - Skip generators when the loop never samples: If your loop never calls
sample, as in SFT or direct preference optimization (DPO), run it on a trainer-only model resource withnum_generator_replicas=0, which costs less.
Use a custom loss function in an RL loop
In a reinforcement learning (RL) loop, a custom loss function replaces only theforward_backward call. Sample completions and score them as usual, then:
- Build one sample per completion, with the prompt and completion as one token sequence and a
weightsrow that’s0over the prompt and1over the completion. Keep each completion’s advantage in your own code, not in the sample. - Run a custom loss step with a loss function that reads the advantages, then call
optim_step. - Call
weights_syncso the nextsamplecall uses the updated policy.
logprobs_of from the first example:
Use the cookbook’s helpers
The cookbook’spost_training_cookbook.custom_loss module makes the forward and custom_forward_backward calls for you, so you write only the loss function. To use it, install the cookbook as a library.
Pass compute_custom_forward_backward a loss function like regularized_sft_loss from the first example. Your loss function receives a list of float32 tensors, one per sample in batch order, each holding that sample’s per-position log-probabilities. It must return a scalar (0-dimensional) tensor, or the helper raises a ValueError. The helper calls .backward() on the result and returns the loss as result.loss:
Next steps
Direct preference optimization
See how DPO snapshots a reference policy and trains with a custom loss function.
Loss functions
Check whether a built-in loss function already fits your objective.
Write a training loop
Create a session, sample completions, and sync weights in the loop around your loss function.
Samples and loss masks
Build the samples a custom loss function reads.