> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Write a custom loss function

> Train toward your own objective with a PyTorch loss over the model's log-probabilities.

A [custom loss function](https://docs.together.ai/docs/post-training/loss-functions.md#when-to-use-a-custom-loss-function) runs in your own process, between two SDK calls. `forward` returns the model's log-probabilities, your PyTorch code computes the loss and its gradients, and `custom_forward_backward` sends the gradients back to the trainer. [How a custom loss function runs](https://docs.together.ai/docs/post-training/loss-functions.md#how-a-custom-loss-function-runs) explains each step.

Read the log-probabilities with `forward`, not with `forward_backward` and `return_loss_fn_outputs=True`. Even with that flag, `forward_backward` runs a backward pass through its built-in loss function, so its gradients add to yours before `optim_step` applies them.

## Run a custom loss step

Build each [sample](https://docs.together.ai/docs/post-training/samples.md) in your `batch` the same way as for supervised fine-tuning (SFT), with `target_tokens` and a `weights` row that marks the positions your loss function reads. The cross-entropy `LossConfig` on `forward` only selects which positions return a log-probability, so every sample needs both arrays, whatever your loss function computes.

Keep any other values your loss function needs, such as reference log-probabilities or advantages, in your own code, not in the sample. Capture them in the loss function, as the example does with `ref`.

This example implements regularized SFT, which adds a penalty to cross-entropy for moving the model's log-probabilities away from a snapshot taken before training:

```python theme={null}
import torch
from together.post_training import AdamParams, LossConfig


def logprobs_of(batch):
    # 1. Read each target token's logprob without accumulating gradients.
    # Positions whose weight is 0 come back as 0.
    scored = session.trainer.forward(
        samples=batch, loss=LossConfig(type="LOSS_TYPE_CROSS_ENTROPY")
    )
    return [
        torch.tensor(out.tensors["logprobs"].data)
        for out in scored.loss_fn_outputs or []
    ]


# Snapshot the reference logprobs once, before training, and read each
# sample's loss mask from its weights row
ref = logprobs_of(batch)
masks = [
    torch.tensor(list(s["loss_fn_inputs"]["weights"]["data"])) for s in batch
]
beta = 0.1


def regularized_sft_loss(logprobs: list[torch.Tensor]) -> torch.Tensor:
    return torch.stack(
        [
            (-lp * m).sum() / m.sum()
            + beta * ((lp - r) ** 2 * m).sum() / m.sum()
            for lp, r, m in zip(logprobs, ref, masks)
        ]
    ).mean()


# 2. Compute the loss and its gradient with respect to every logprob
logprobs = [lp.requires_grad_() for lp in logprobs_of(batch)]
loss = regularized_sft_loss(logprobs)
gradients = torch.autograd.grad(loss, logprobs)

# 3. Send the gradients back, then apply them
session.trainer.custom_forward_backward(
    samples=batch, gradients=[{"data": grad.tolist()} for grad in gradients]
)
session.trainer.optim_step(
    adam_params=AdamParams(learning_rate=1e-6, weight_decay=0.1),
)

# The service doesn't report a loss for a custom step, so log it yourself
print(loss.item())
```

Get these details right:

* **Average over the trained positions only:** `forward` returns a real log-probability only where the sample's `weights` entry is non-zero, and `0.0` at every other position. The service ignores gradients at those zero-weight positions, but the zeros still count in your loss if you average over the whole row. Divide by the mask's sum instead, as the example does with `m.sum()`.
* **Match each gradient list to its sample:** Pass `custom_forward_backward` one gradient list per sample, in batch order. Each list must have one value per target position, the same length as the sample's `target_tokens`.
* **Snapshot a frozen reference once:** Call `forward` for the reference before training starts. Recomputing the reference every step repeats the same forward passes.
* **Split a large batch across `forward` calls:** `forward` scores the whole batch in one operation. If it runs out of memory or returns no log-probabilities, call it on smaller slices of the batch. If the call times out, use smaller slices or pass a longer `timeout`.
* **Skip generators when the loop never samples:** If your loop never calls `sample`, as in SFT or direct preference optimization (DPO), run it on a [trainer-only model resource](https://docs.together.ai/docs/post-training/model-resources.md#generator-replicas) with `num_generator_replicas=0`, which costs less.

## Use a custom loss function in an RL loop

In a reinforcement learning (RL) loop, a custom loss function replaces only the `forward_backward` call. Sample completions and score them as usual, then:

1. Build one sample per completion, with the prompt and completion as one token sequence and a `weights` row that's `0` over the prompt and `1` over the completion. Keep each completion's advantage in your own code, not in the sample.
2. [Run a custom loss step](#run-a-custom-loss-step) with a loss function that reads the advantages, then call `optim_step`.
3. Call `weights_sync` so the next `sample` call uses the updated policy.

This example implements a plain policy-gradient objective, which you replace with your own. It reuses `logprobs_of` from the first example:

```python theme={null}
import torch
from together.post_training import AdamParams

# batch has one sample per completion, and advantages one float per sample
masks = [
    torch.tensor(list(s["loss_fn_inputs"]["weights"]["data"])) for s in batch
]


def policy_gradient_loss(logprobs: list[torch.Tensor]) -> torch.Tensor:
    return torch.stack(
        [
            -(adv * lp * m).sum() / m.sum()
            for lp, m, adv in zip(logprobs, masks, advantages)
        ]
    ).mean()


logprobs = [lp.requires_grad_() for lp in logprobs_of(batch)]
gradients = torch.autograd.grad(policy_gradient_loss(logprobs), logprobs)
session.trainer.custom_forward_backward(
    samples=batch, gradients=[{"data": grad.tolist()} for grad in gradients]
)
session.trainer.optim_step(adam_params=AdamParams(learning_rate=1e-6))
session.trainer.weights_sync()
```

## Use the cookbook's helpers

The cookbook's `post_training_cookbook.custom_loss` module makes the `forward` and `custom_forward_backward` calls for you, so you write only the loss function. To use it, [install the cookbook as a library](https://docs.together.ai/docs/post-training/cookbook-overview.md#install-as-a-library).

Pass `compute_custom_forward_backward` a loss function like `regularized_sft_loss` from the first example. Your loss function receives a list of `float32` tensors, one per sample in batch order, each holding that sample's per-position log-probabilities. It must return a scalar (0-dimensional) tensor, or the helper raises a `ValueError`. The helper calls `.backward()` on the result and returns the loss as `result.loss`:

```python theme={null}
import torch

from post_training_cookbook.custom_loss import (
    compute_custom_forward_backward,
    compute_logprobs,
)
from together.post_training import AdamParams

# Snapshot the reference logprobs with compute_logprobs, which runs
# only the forward pass
ref = [torch.tensor(r) for r in compute_logprobs(session, batch)]

# One training step with regularized_sft_loss from the first example
result = compute_custom_forward_backward(session, batch, regularized_sft_loss)
session.trainer.optim_step(
    adam_params=AdamParams(learning_rate=1e-6, weight_decay=0.1),
)
print(result.loss)
```

## Next steps

> [card] [Direct preference optimization](https://docs.together.ai/docs/post-training/dpo.md)
>
> See how DPO snapshots a reference policy and trains with a custom loss function.

> [card] [Loss functions](https://docs.together.ai/docs/post-training/loss-functions.md#built-in-loss-functions)
>
> Check whether a built-in loss function already fits your objective.

> [card] [Write a training loop](https://docs.together.ai/docs/post-training/training-loop.md)
>
> Create a session, sample completions, and sync weights in the loop around your loss function.

> [card] [Samples and loss masks](https://docs.together.ai/docs/post-training/samples.md)
>
> Build the samples a custom loss function reads.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.