Skip to main content
A custom loss function runs in your own process, between two SDK calls. forward returns the model’s log-probabilities, your PyTorch code computes the loss and its gradients, and custom_forward_backward sends the gradients back to the trainer. How a custom loss function runs explains each step. Read the log-probabilities with forward, not with forward_backward and return_loss_fn_outputs=True. Even with that flag, forward_backward runs a backward pass through its built-in loss function, so its gradients add to yours before optim_step applies them.

Run a custom loss step

Build each sample in your batch the same way as for supervised fine-tuning (SFT), with target_tokens and a weights row that marks the positions your loss function reads. The cross-entropy LossConfig on forward only selects which positions return a log-probability, so every sample needs both arrays, whatever your loss function computes. Keep any other values your loss function needs, such as reference log-probabilities or advantages, in your own code, not in the sample. Capture them in the loss function, as the example does with ref. This example implements regularized SFT, which adds a penalty to cross-entropy for moving the model’s log-probabilities away from a snapshot taken before training:
Get these details right:
  • Average over the trained positions only: forward returns a real log-probability only where the sample’s weights entry is non-zero, and 0.0 at every other position. The service ignores gradients at those zero-weight positions, but the zeros still count in your loss if you average over the whole row. Divide by the mask’s sum instead, as the example does with m.sum().
  • Match each gradient list to its sample: Pass custom_forward_backward one gradient list per sample, in batch order. Each list must have one value per target position, the same length as the sample’s target_tokens.
  • Snapshot a frozen reference once: Call forward for the reference before training starts. Recomputing the reference every step repeats the same forward passes.
  • Split a large batch across forward calls: forward scores the whole batch in one operation. If it runs out of memory or returns no log-probabilities, call it on smaller slices of the batch. If the call times out, use smaller slices or pass a longer timeout.
  • Skip generators when the loop never samples: If your loop never calls sample, as in SFT or direct preference optimization (DPO), run it on a trainer-only model resource with num_generator_replicas=0, which costs less.

Use a custom loss function in an RL loop

In a reinforcement learning (RL) loop, a custom loss function replaces only the forward_backward call. Sample completions and score them as usual, then:
  1. Build one sample per completion, with the prompt and completion as one token sequence and a weights row that’s 0 over the prompt and 1 over the completion. Keep each completion’s advantage in your own code, not in the sample.
  2. Run a custom loss step with a loss function that reads the advantages, then call optim_step.
  3. Call weights_sync so the next sample call uses the updated policy.
This example implements a plain policy-gradient objective, which you replace with your own. It reuses logprobs_of from the first example:

Use the cookbook’s helpers

The cookbook’s post_training_cookbook.custom_loss module makes the forward and custom_forward_backward calls for you, so you write only the loss function. To use it, install the cookbook as a library. Pass compute_custom_forward_backward a loss function like regularized_sft_loss from the first example. Your loss function receives a list of float32 tensors, one per sample in batch order, each holding that sample’s per-position log-probabilities. It must return a scalar (0-dimensional) tensor, or the helper raises a ValueError. The helper calls .backward() on the result and returns the loss as result.loss:

Next steps

Direct preference optimization

See how DPO snapshots a reference policy and trains with a custom loss function.

Loss functions

Check whether a built-in loss function already fits your objective.

Write a training loop

Create a session, sample completions, and sync weights in the loop around your loss function.

Samples and loss masks

Build the samples a custom loss function reads.