Choose a training method
Choose a method based on the training data you can produce:
The methods also differ in what they can teach and what they cost:
- SFT: It’s the simplest method to run, because it needs no reward function. It can teach only what your demonstrations show. To improve accuracy beyond them on a task you can score, continue training with GRPO.
- DPO: Use it for behavior that’s easier to judge than to demonstrate, where you can say which of two responses is better but can’t write the ideal one.
- GRPO: Use it for tasks where your code can check an answer, such as math with a known solution or code with tests, and for agents that act over several turns and call tools. It’s the only method that samples while it trains, so its model resource needs at least one generator replica, which adds GPUs and cost. SFT and DPO run on trainer-only resources.
Combine methods
To run methods in sequence, hand off through a training checkpoint. Each phase runs in its own session, typically on a new model resource for the same base model, because a GRPO phase needs a generator replica. Keep the base model and the training mode the same in every phase. A training checkpoint from a LoRA session can’t start a full-weight session, or the reverse. The handoff takes three steps:- Save a training checkpoint at the last step of the first run.
- Start the next session from that checkpoint with
load_optimizer=False. The new session loads the adapter weights with a fresh optimizer and its step counter at zero. - Use the same LoRA rank and alpha in both phases. A different rank makes the checkpoint fail to load, and a different alpha silently changes how strongly the adapter applies.