> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage model resources

> Configure, create, attach to, inspect, and stop post-training model resources from the SDK or the console.

A *model resource* represents the set of GPUs allocated for training or sampling, plus the base model loaded onto them. Every [session](https://docs.together.ai/docs/post-training/concepts.md#model-resources-and-sessions) runs on a model resource, and Together AI bills you for the resource, not its sessions.

This page covers how to choose a configuration, create a resource or attach to an existing one, inspect its state, and stop it, from the SDK or the Together AI console. For lifecycle states, timers, error codes, and limits, see [Lifecycle, errors, and limits](https://docs.together.ai/docs/post-training/limits-and-errors.md).

If you train with a cookbook recipe, the recipe manages the resource for you. Set `model` in the recipe's config, and the recipe creates a resource and stops it on exit. In group relative policy optimization (GRPO) and supervised fine-tuning (SFT) recipes, you can instead set `model_resources_id` to attach to a resource that's already running.

## Choose a configuration

A model resource's settings are fixed when you create it, and every session on it shares them. To change one, create a new resource. In the SDK, pass the settings to `ModelResourcesClient.create`. For the full request schema, see the [API reference](https://docs.together.ai/reference/post-training/model-resources-create.md).

### Base model

Set `base_model` in the SDK, or `model` in a cookbook config, to the ID of a [supported model](https://docs.together.ai/docs/post-training/supported-models.md).

To train your own weights in place of the base model's, also pass `base_weights_ref`, pointing to a model in your project's model registry as `together://<object_id>` or `together://<object_id>@<revision_id>`. It must be a full, unquantized model with the base model's architecture. `base_model` still selects the compute layout. Checkpoints resume only on resources created with the same weights. Cookbook configs don't expose `base_weights_ref`.

Qwen3.8-27B inference checkpoints are saved in FP8, so you can't use them as base weights. To keep training a model from a Qwen3.8-27B session, resume from the session's [training checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#resume-or-fork-a-run) instead, if you saved one.

### Training mode

Set `lora_enabled` in the SDK, or `use_lora` in a cookbook config. Both default to `True`, which trains a low-rank adapter (LoRA). A LoRA resource hosts up to five adapter sessions at once, or fewer for some models. Set the flag to `False` for full-weight training, which limits the resource to one session at a time. Each new session on a full-weight resource starts from the base weights, or from the [checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#resume-or-fork-a-run) it resumes, never from an earlier session's weights. It also takes longer to start, because the model is reloaded. To choose between the two modes, see [LoRA vs. full-weight training](https://docs.together.ai/docs/post-training/lora-vs-full-weight.md).

### GPU type

Post-training currently runs only on [B200 GPUs](https://docs.together.ai/docs/post-training/supported-models.md#choose-a-gpu-type), so set `gpu_type="B200-SXM"` in `ComputeConfig`. Cookbook configs don't expose the GPU type.

### Generator replicas

By default, a model resource runs a trainer and one generator replica. The trainer computes losses and gradients and applies optimizer steps. Generators sample completions for reinforcement learning or for evaluating saved checkpoints.

Set the generator count with `num_generator_replicas` in `ComputeConfig`, or in `GRPOConfig` for a GRPO recipe. Both default to one replica, even when you pass a `compute_config` that sets only `gpu_type`.

* **SFT and direct preference optimization (DPO):** Set `num_generator_replicas=0` explicitly to make the resource trainer-only. A trainer-only resource provisions less hardware and costs less per minute. The cookbook's SFT and DPO loops do this for you.
* **GRPO:** Set `num_generator_replicas` to 1 or more. If rollouts are the bottleneck rather than the optimizer step, add replicas to raise sampling throughput. Each replica adds GPUs and cost.

Models whose row in the [catalog](https://docs.together.ai/docs/post-training/supported-models.md#catalog) says **No generator** can't run generator replicas, so creating a resource for one fails unless you set `num_generator_replicas=0`.

Sessions inherit the resource's layout. On a trainer-only resource, `session.generator` raises `RuntimeError`, so in code that runs on resources with and without generators, check `session.has_generator` first.

### Optimizer

Resources use Adam by default. To use Muon, which requires full-weight training, pass `optimizer_config=OptimizerConfig(muon=MuonConfig())` in the SDK, or set `optimizer="muon"` in `GRPOConfig`. Muon falls back to Adam for embeddings, norm layers, and the output head.

The learning rate isn't a resource setting. Pass it on every `optim_step` instead.

### Sampling-only resources

To sample a saved [inference checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#inference-checkpoints) without training it further, create a resource with no trainer by setting `num_trainer_replicas=0` in `ComputeConfig`. The resource provisions only its generator GPUs, so its [cost estimate](https://docs.together.ai/docs/post-training/pricing.md#estimate-the-cost-of-a-resource) and its bill cover only those.

A sampling-only resource has these requirements:

* **Generators:** Set `num_generator_replicas` to 1 or more.
* **Optimizer:** Omit `optimizer_config`.
* **Training mode:** Set `lora_enabled` to `True` to load the checkpoint's LoRA adapter, or to `False` to load its full model weights.

Every session on a sampling-only resource [starts from an inference checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#sample-an-inference-checkpoint) and can only sample.

## Create a resource

To create a resource, call `ModelResourcesClient.create`. Your project needs [enough credit to run it for four hours](https://docs.together.ai/docs/post-training/pricing.md#minimum-credit-balance) at its hourly rate.

`create` submits the request, then waits while the resource queues for GPUs, starts them, and loads the base model. It returns once the resource is `READY`, which is when billing starts.

```python theme={null}
from together.post_training import ComputeConfig, ModelResourcesClient

with ModelResourcesClient.create(
    base_model="Qwen/Qwen3.8-27B",
    lora_enabled=True,
    compute_config=ComputeConfig(
        gpu_type="B200-SXM",
        num_generator_replicas=1,
    ),
) as resources:
    print(resources.model_resources_id)
    with resources.create_session() as session:
        ...  # your training loop
```

Used as a context manager, the handle stops the resource on exit, even if your loop raises. The exit passes `force=True`, so every session still attached to the resource stops with it, including sessions that other processes own. Nest the session's context manager inside it so your session stops first.

How long `create` waits depends on free capacity for the model's GPU type. Neither the service nor the SDK limits how long a resource waits in the queue, so by default `create` keeps waiting until the resource is allocated the required GPUs, printing its queue position as it goes. To give up sooner, pass `ready_timeout` in seconds.

If creation fails (the wait times out, the resource enters `ERROR`, or you interrupt the process), the SDK stops the resource it created before raising, so a failed create doesn't leave GPUs billing. If that stop fails too, the SDK raises the stop's error, with the create failure chained to it. The resource might still be billing, so [stop it yourself](#stop-a-resource).

If the process is killed instead, for example with `kill -9` or by a machine crash, no cleanup runs, so [stop the resource yourself](#stop-a-resource). A resource that's still queued when the process dies keeps waiting until it gets GPUs, then becomes `READY` and bills until you stop it or it reaches its [idle timeout](#idle-timeout).

In [async code](https://docs.together.ai/docs/post-training/training-loop.md#call-operations-asynchronously), use `ModelResourcesClient.create_async` instead.

You can also create a resource from the **Model resources** tab under [Post-Training](https://api.together.ai/post-training/resources) in the console, then [attach sessions to it](#attach-to-an-existing-resource) from code.

## Attach to an existing resource

To train on a resource that already exists, attach to it by ID. Use this for a resource you created in the console, one a teammate provisioned, or one you keep running across experiments. `attach` provisions nothing and doesn't wait. It only confirms that the resource exists.

> [!NOTE]
>
> An attached handle doesn't own the resource, so it never stops the resource for you. Used as a context manager, it calls `detach()` on exit and leaves the resource running. Without the context manager, call `detach()` when you're done. To stop the resource, call `stop()`.
>
> Detaching on exit requires SDK version 3.2.0 or later. In earlier versions, the context manager stops the resource on exit.

```python theme={null}
from together.post_training import ModelResourcesClient

resources = ModelResourcesClient.attach(
    model_resources_id="your_model_resources_id"
)
try:
    with resources.create_session() as session:
        ...  # your training loop
finally:
    resources.detach()  # releases the handle, leaves the resource running
```

To [attach a GRPO or SFT cookbook recipe](https://docs.together.ai/docs/post-training/multi-lora.md#attach-to-a-running-resource) instead, set `model_resources_id` in its config.

## Inspect and list resources

To check a resource's state, call `retrieve` on its handle. It returns the resource's current record, including its `status` and, when the status is `ERROR`, an `error` with the error code:

```python theme={null}
resource = resources.retrieve()
print(resource.status, resource.compute_config.gpu_type)
```

While a resource waits in the queue, `resource.status_details.queue_position` estimates its place in line, counting the older resources of the same GPU type still waiting for GPUs, plus one. It's unset once the resource is placed on a cluster.

To find resources without a handle, list them from the base client. Filter by status to see what's billing right now:

```python theme={null}
from together import Together

client = Together()
for r in client.post_training.model_resources.list(
    status=["MODEL_RESOURCES_STATUS_READY"]
).data:
    print(
        r.id,
        r.base_model,
        r.compute_config.num_generator_replicas,
        r.created_by,
    )
```

To find resources in several states, call `list` once for each status. `status` takes a list, but the SDK sends several values in a form the API rejects with HTTP `400`.

The **Model resources** tab in the [console](#manage-resources-in-the-console) shows the same resources. The Together CLI lists and inspects them with `tg training ls-resources`, `tg training ls-sessions`, `tg training ls-checkpoints`, and `tg training get <ID>`. It can't create or stop resources, so use the SDK or the console for that.

## Stop a resource

A `READY` resource bills until it stops, even when nothing is training on it. It [stops on its own](https://docs.together.ai/docs/post-training/limits-and-errors.md#timers) when it reaches its [idle timeout](#idle-timeout) (one hour by default) without an operation, or when the billing system reports that your project's credit limit has been exceeded or your organization is blocked. Stop it yourself as soon as you're done:

```python theme={null}
resources.stop()
```

`stop` returns as soon as the API accepts the stop, which is when billing ends.

The API refuses to stop a resource while any session on it is still active. Stop your sessions first, or pass `force=True` to stop the resource along with every session attached to it, including sessions that other processes own:

```python theme={null}
resources.stop(force=True)
```

To stop a resource you don't have a handle for, such as one an interrupted run left behind, call `stop` on the base client with the resource's ID:

```python theme={null}
from together import Together

client = Together()
client.post_training.model_resources.stop(
    "your_model_resources_id", force=True
)
```

You can also stop any running resource from the **Model resources** tab in the [console](#manage-resources-in-the-console).

## Idle timeout

A resource's *idle timeout* is how long it stays `READY` without activity before it stops. Becoming `READY`, starting a session, and every operation on any of its sessions count as activity. A resource doesn't stop while an operation is running.

Set the timeout with `inactive_timeout` when you create the resource, either as an argument to `ModelResourcesClient.create` or as a field in the [API request](https://docs.together.ai/reference/post-training/model-resources-create.md). The value is in seconds, from 300 (five minutes) to 86400 (24 hours), and defaults to 3600 (one hour). You can't change it after the resource is created.

The API returns each resource with its `inactive_timeout` and, while it's `READY`, an `expires_at` timestamp. `expires_at` is the earliest time the resource can stop for inactivity, which is its last activity plus `inactive_timeout`. It refreshes about once a minute and never moves earlier. The resource's details page in the [console](#manage-resources-in-the-console) shows the same time as **Stops if idle at**.

To keep a resource `READY` while you aren't running operations, for example while you compute rewards, call `extend_expiry` on its handle. Pass `seconds`, from 60 to 86400, and the resource doesn't stop for inactivity until at least that many seconds from now:

```python theme={null}
resource = resources.extend_expiry(seconds=7200)
print(resource.expires_at)
```

The call returns the resource with its new `expires_at`. It only moves `expires_at` later, so passing fewer seconds than the resource already has left changes nothing. If the resource isn't `READY` or `seconds` is out of range, the call fails with HTTP `400`, which the SDK raises as `BadRequestError`.

To extend the expiry of a resource you don't have a handle for, call `client.post_training.model_resources.extend_expiry("your_model_resources_id", seconds=7200)` from the base client.

## Manage resources in the console

To create or stop a resource without code, open the **Model resources** tab under [Post-Training](https://api.together.ai/post-training/resources) in the console. The tab lists your active resources with their status, hourly rate, and estimated spend. Your past resources appear below them. Select **Create model resource** to create one, or stop any resource that's still running. Statuses match the API's [lifecycle states](https://docs.together.ai/docs/post-training/limits-and-errors.md#lifecycle).

Select a resource's base model name to open its details page. While the resource is `READY`, the page shows **Stops if idle at**: the earliest time, in your local time zone, that the resource can stop if no session runs an operation before then, followed by its idle timeout. The time comes from the resource's [`expires_at`](#idle-timeout), so it moves later with each operation and each `extend_expiry` call.

The console also [shows a session's progress and its resource's cost](https://docs.together.ai/docs/post-training/monitoring.md#watch-a-run-in-the-console) while a run trains.

## Next steps

> [card] [Train multiple adapters](https://docs.together.ai/docs/post-training/multi-lora.md)
>
> Share one resource across several sessions, so a whole sweep runs on one base model's worth of GPUs.

> [card] [Pricing](https://docs.together.ai/docs/post-training/pricing.md)
>
> Learn how a resource bills, how to estimate a run's cost, and what's refunded on failure.

> [card] [Post-training API reference](https://docs.together.ai/reference/post-training/model-resources-create.md)
>
> Look up every field for resources, sessions, and operations, with its type and default.

> [card] [Supported models](https://docs.together.ai/docs/post-training/supported-models.md)
>
> Check the GPU types, training modes, and session caps for each model.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.