> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage checkpoints

> Save inference and training checkpoints, download trained weights, resume or fork a run, and find checkpoints after a session ends.

![The two kinds of checkpoints a training session produces. A training checkpoint holds the full training state, and you use it to fork a new session or resume the same run. An inference checkpoint holds adapter or merged weights, and you use it to download the weights or deploy them to production.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/checkpoint-types.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=8f9a51f1c58904b8289c63ef921b979f)

A training session produces two kinds of checkpoints:

* An **inference checkpoint** snapshots the model weights, ready to download. For most models, it also registers them in the model registry so you can [deploy](https://docs.together.ai/docs/post-training/deployment.md) them.
* A **training checkpoint** saves the full training state so a later session can resume or fork the run.

## Inference checkpoints

Cookbook recipes save an inference checkpoint at the final step, and every `inference_checkpoint_interval` steps if you set it. To save one from the SDK, call `create_inference_checkpoint`:

```python theme={null}
checkpoint = session.create_inference_checkpoint()
print(checkpoint.id, checkpoint.step)
```

The call returns an `InferenceCheckpoint` with an `id`, the `step` it was saved at, and a `registration` that holds its model registry IDs. The session also lists the checkpoint in `inference_checkpoints`.

A checkpoint from a low-rank adapter (LoRA) session registers the adapter in `registration.adapter`. A checkpoint from a full-weight session, or from a LoRA session on custom base weights, registers a merged full model in `registration.model` instead. Each holds a model registry `id` (`ml_...`) and a `revision_id` (`rv_...`), which you use to [deploy the checkpoint](https://docs.together.ai/docs/post-training/deployment.md).

A checkpoint that wasn't uploaded to the model registry has no `registration`, so it can't be deployed. Checkpoints on models marked **Not registered** in [Supported models](https://docs.together.ai/docs/post-training/supported-models.md#checkpoints-and-deployment) aren't uploaded, because dedicated model inference can't serve their adapters. [Download them](#download-the-weights) instead.

### Download the weights

To download an inference checkpoint's files to a local directory, call `download_checkpoint` with the checkpoint's `id`. Training checkpoints can't be downloaded, only resumed or forked.

Pass `variant` to choose which files you get:

* `CHECKPOINT_VARIANT_MERGED` (the default): The base model with the adapter merged in, ready for any inference framework. Available for full-weight sessions, and for LoRA sessions on models that support it.
* `CHECKPOINT_VARIANT_ADAPTER`: The adapter weights alone, which are much smaller but need the base model alongside them for inference. On models marked **Adapter only** in [Supported models](https://docs.together.ai/docs/post-training/supported-models.md#checkpoints-and-deployment), pass this variant.

The adapter files use SGLang's multi-LoRA layout, not the PEFT layout, so PEFT and Hugging Face Transformers can't load them. To run a trained model in another inference framework, download `CHECKPOINT_VARIANT_MERGED`:

```python theme={null}
from pathlib import Path

from together import Together
from together.post_training import download_checkpoint

client = Together()
paths = download_checkpoint(
    client,
    "<inference_checkpoint_id>",
    variant="CHECKPOINT_VARIANT_MERGED",
    output_dir=Path("./trained_model"),
)
for path in paths:
    print(f"  {path.name}  ({path.stat().st_size:,} bytes)")
```

You should see the standard Hugging Face layout:

```text theme={null}
  chat_template.jinja  (8,952 bytes)
  config.json  (3,636 bytes)
  ...
  model-00001-of-00018.safetensors  (3,966,730,520 bytes)
  model-00002-of-00018.safetensors  (3,043,080,296 bytes)
  ...
  tokenizer.json  (12,809,320 bytes)
  ...
```

For a Qwen3.8-27B session, whether full-weight or LoRA, the merged weights are in the block FP8 format of `Qwen/Qwen3.8-27B-FP8`, the model that dedicated model inference serves.

## Training checkpoints

A training checkpoint saves the trained weights (the adapter, for a LoRA session), the optimizer state, and the step counter.

To have a cookbook recipe save training checkpoints, set `training_checkpoint_interval` on its config. The cookbook saves one only every `training_checkpoint_interval` steps, never automatically at the final step. To get one at the end of the run, set the interval equal to the step count. The cookbook records the checkpoint IDs in `runs/<run_name>/checkpoints.jsonl`.

To save a training checkpoint from the SDK, call `create_training_checkpoint`:

```python theme={null}
checkpoint = session.create_training_checkpoint()
print(checkpoint.id, checkpoint.step)
```

### Resume or fork a run

To start a session from a training checkpoint, pass its ID as `resume_from_checkpoint_id` to `resources.create_session`, `SessionClient.create`, or any cookbook config. All of them also take `load_optimizer`, which applies only when you pass a checkpoint ID. It decides whether the new session continues the original run or starts a new one:

* **Continue the same run:** Omit `load_optimizer` or pass `True`. The new session restores the adapter, the optimizer state, and the step counter, and training continues where it left off. For a cookbook group relative policy optimization (GRPO) run, [continue the run from its recipe](https://docs.together.ai/docs/post-training/grpo-recipes.md#continue-a-run) to also pick up its data and learning-rate schedule where they stopped.
* **Start a new run from the weights:** Pass `load_optimizer=False`. The new session loads only the adapter weights, with a fresh optimizer and a step counter at zero. Use this to fork from a known-good state and try different hyperparameters, or to [hand off from one training method to another](https://docs.together.ai/docs/post-training/training-methods.md#combine-methods), such as from supervised fine-tuning (SFT) to GRPO.

In both cases, the new session must be compatible with the checkpoint:

* **LoRA settings:** Pass the same `lora_config` (rank and alpha) the original session used. The session doesn't inherit LoRA settings from the checkpoint, so if you omit `rank`, `alpha`, or `dropout`, the server applies its defaults. The exception is `train_unembed`, which the session takes from the checkpoint when you omit it. The API rejects a `train_unembed` value that differs from the checkpoint's.
* **Base weights:** The model resource must use the same base weights as the resource that saved the checkpoint. Catalog entries that share weights are interchangeable, so a checkpoint saved on `Qwen/Qwen3.8-27B-single-session` resumes on `Qwen/Qwen3.8-27B`.
* **Checkpoint type:** On a resource with a trainer, LoRA sessions resume from training checkpoints only. To [sample an inference checkpoint](#sample-an-inference-checkpoint) instead, start a session from it on a sampling-only resource.

This example starts a new run from a checkpoint's weights:

```python theme={null}
from together.post_training import LoraConfig, ModelResourcesClient

with ModelResourcesClient.create(
    base_model="Qwen/Qwen3.8-27B",
    compute_config={"gpu_type": "B200-SXM"},
) as resources:
    with resources.create_session(
        resume_from_checkpoint_id="<training_checkpoint_id>",
        load_optimizer=False,
        lora_config=LoraConfig(rank=16, alpha=16),
    ) as session:
        ...  # continue training
```

## Find a checkpoint

To look up checkpoints after a session ends, call `client.post_training.checkpoints.list`. It returns your project's training and inference checkpoints, newest first. Pass `session_id`, `base_model`, or `type` to filter them.

For example, to recover a training checkpoint ID when you no longer have the run directory's `checkpoints.jsonl`, list the session's training checkpoints:

```python theme={null}
from together import Together

client = Together()
page = client.post_training.checkpoints.list(
    session_id="<session_id>",
    type="CHECKPOINT_TYPE_TRAINING",
)
for checkpoint in page.data:
    print(checkpoint.id, checkpoint.step)
```

Omit `type` to get both kinds, or pass `CHECKPOINT_TYPE_INFERENCE` to get only inference checkpoints.

Each call returns one page of at most `limit` checkpoints (1 to 100). When `page.meta.has_more` is true, pass `page.meta.next_cursor` as `after` to get the next page.

A listed inference checkpoint holds its model registry IDs in `inference_registration.adapter` or `inference_registration.model`. A listed training checkpoint keeps them in `training_registration`, which holds the training state and can't be deployed. To resume, pass the checkpoint's `id`, not these registry IDs.

To get one checkpoint's details, call `client.post_training.checkpoints.retrieve("<checkpoint_id>")`. Its `lora_rank` is the rank to pass in `lora_config` when you [resume or fork a run](#resume-or-fork-a-run) from it. Checkpoints from full-weight sessions have no `lora_rank`.

## Sample a saved checkpoint

A generator samples only from its own session's current weights, so you can't sample a saved checkpoint directly. Choose a route based on what you want to sample:

* **Training checkpoint:** [Start a session from it](#resume-or-fork-a-run) with `resume_from_checkpoint_id` on a model resource that has a trainer and at least one generator replica. The service loads the checkpoint's weights onto the generator when the session starts, so `session.generator.sample` returns completions from the saved model before you run any training step.
* **Inference checkpoint:** [Start a session from it](#sample-an-inference-checkpoint) on a sampling-only model resource. If it has a `registration`, you can also [deploy it](https://docs.together.ai/docs/post-training/deployment.md) to dedicated model inference and send requests to the deployment.
* **Base model:** If [serverless inference](https://docs.together.ai/docs/serverless/models.md) hosts the base model, call it there, the same way you [check a baseline before training](https://docs.together.ai/docs/post-training/evaluate.md#check-a-baseline-before-training).

### Sample an inference checkpoint

To sample an inference checkpoint without deploying it, start a session from it on a [sampling-only resource](https://docs.together.ai/docs/post-training/model-resources.md#sampling-only-resources), one created with `num_trainer_replicas=0`. The checkpoint must belong to your project, and the resource must use the same base weights as the resource that saved the checkpoint.

The generators load the checkpoint's weights when the session starts. Which weights they load depends on the resource's training mode:

* **LoRA resource:** The session loads the checkpoint's adapter and takes its LoRA settings from the checkpoint. The checkpoint must hold an adapter.
* **Full-weight resource:** The session loads the checkpoint's full model weights. They must match the base model's architecture and quantization, so an FP8 export doesn't load onto a BF16 base model. A checkpoint that holds only an adapter can't start a session here.

To start the session, pass the checkpoint's ID as `resume_from_checkpoint_id`, and omit `lora_config` and `load_optimizer`:

```python theme={null}
from together.post_training import ComputeConfig, ModelResourcesClient

with ModelResourcesClient.create(
    base_model="Qwen/Qwen3.8-27B",
    lora_enabled=True,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM",
        num_trainer_replicas=0,
        num_generator_replicas=1,
    ),
) as resources:
    with resources.create_session(
        resume_from_checkpoint_id="<inference_checkpoint_id>",
    ) as session:
        ...  # sample with session.generator.sample
```

The session's weights stay fixed, and it accepts only [sample](https://docs.together.ai/docs/post-training/training-loop.md#sample-from-the-generator) operations, not training steps, weight syncs, or checkpoint saves.

## Next steps

> [card] [Deploy a checkpoint](https://docs.together.ai/docs/post-training/deployment.md)
>
> Serve a trained model on dedicated model inference.

> [card] [Combine methods](https://docs.together.ai/docs/post-training/training-methods.md#combine-methods)
>
> Hand a training checkpoint from one method to the next, such as SFT to GRPO.

> [card] [Lifecycle, errors, and limits](https://docs.together.ai/docs/post-training/limits-and-errors.md#timers)
>
> Learn the timers and stops that can end a run, and how training checkpoints limit what you lose.

> [card] [Start a training session](https://docs.together.ai/reference/post-training/training-sessions-create.md)
>
> Look up every field a session takes when it resumes from a checkpoint.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.