model in the recipe’s config, and the recipe creates a resource and stops it on exit. In group relative policy optimization (GRPO) and supervised fine-tuning (SFT) recipes, you can instead set model_resources_id to attach to a resource that’s already running.
Choose a configuration
A model resource’s settings are fixed when you create it, and every session on it shares them. To change one, create a new resource. In the SDK, pass the settings toModelResourcesClient.create. For the full request schema, see the API reference.
Base model
Setbase_model in the SDK, or model in a cookbook config, to the ID of a supported model.
To train your own weights in place of the base model’s, also pass base_weights_ref, pointing to a model in your project’s model registry as together://<object_id> or together://<object_id>@<revision_id>. It must be a full, unquantized model with the base model’s architecture. base_model still selects the compute layout. Checkpoints resume only on resources created with the same weights. Cookbook configs don’t expose base_weights_ref.
Qwen3.8-27B inference checkpoints are saved in FP8, so you can’t use them as base weights. To keep training a model from a Qwen3.8-27B session, resume from the session’s training checkpoint instead, if you saved one.
Training mode
Setlora_enabled in the SDK, or use_lora in a cookbook config. Both default to True, which trains a low-rank adapter (LoRA). A LoRA resource hosts up to five adapter sessions at once, or fewer for some models. Set the flag to False for full-weight training, which limits the resource to one session at a time. Each new session on a full-weight resource starts from the base weights, or from the checkpoint it resumes, never from an earlier session’s weights. It also takes longer to start, because the model is reloaded. To choose between the two modes, see LoRA vs. full-weight training.
GPU type
Post-training currently runs only on B200 GPUs, so setgpu_type="B200-SXM" in ComputeConfig. Cookbook configs don’t expose the GPU type.
Generator replicas
By default, a model resource runs a trainer and one generator replica. The trainer computes losses and gradients and applies optimizer steps. Generators sample completions for reinforcement learning or for evaluating saved checkpoints. Set the generator count withnum_generator_replicas in ComputeConfig, or in GRPOConfig for a GRPO recipe. Both default to one replica, even when you pass a compute_config that sets only gpu_type.
- SFT and direct preference optimization (DPO): Set
num_generator_replicas=0explicitly to make the resource trainer-only. A trainer-only resource provisions less hardware and costs less per minute. The cookbook’s SFT and DPO loops do this for you. - GRPO: Set
num_generator_replicasto 1 or more. If rollouts are the bottleneck rather than the optimizer step, add replicas to raise sampling throughput. Each replica adds GPUs and cost.
num_generator_replicas=0.
Sessions inherit the resource’s layout. On a trainer-only resource, session.generator raises RuntimeError, so in code that runs on resources with and without generators, check session.has_generator first.
Optimizer
Resources use Adam by default. To use Muon, which requires full-weight training, passoptimizer_config=OptimizerConfig(muon=MuonConfig()) in the SDK, or set optimizer="muon" in GRPOConfig. Muon falls back to Adam for embeddings, norm layers, and the output head.
The learning rate isn’t a resource setting. Pass it on every optim_step instead.
Sampling-only resources
To sample a saved inference checkpoint without training it further, create a resource with no trainer by settingnum_trainer_replicas=0 in ComputeConfig. The resource provisions only its generator GPUs, so its cost estimate and its bill cover only those.
A sampling-only resource has these requirements:
- Generators: Set
num_generator_replicasto 1 or more. - Optimizer: Omit
optimizer_config. - Training mode: Set
lora_enabledtoTrueto load the checkpoint’s LoRA adapter, or toFalseto load its full model weights.
Create a resource
To create a resource, callModelResourcesClient.create. Your project needs enough credit to run it for four hours at its hourly rate.
create submits the request, then waits while the resource queues for GPUs, starts them, and loads the base model. It returns once the resource is READY, which is when billing starts.
force=True, so every session still attached to the resource stops with it, including sessions that other processes own. Nest the session’s context manager inside it so your session stops first.
How long create waits depends on free capacity for the model’s GPU type. Neither the service nor the SDK limits how long a resource waits in the queue, so by default create keeps waiting until the resource is allocated the required GPUs, printing its queue position as it goes. To give up sooner, pass ready_timeout in seconds.
If creation fails (the wait times out, the resource enters ERROR, or you interrupt the process), the SDK stops the resource it created before raising, so a failed create doesn’t leave GPUs billing. If that stop fails too, the SDK raises the stop’s error, with the create failure chained to it. The resource might still be billing, so stop it yourself.
If the process is killed instead, for example with kill -9 or by a machine crash, no cleanup runs, so stop the resource yourself. A resource that’s still queued when the process dies keeps waiting until it gets GPUs, then becomes READY and bills until you stop it or it reaches its idle timeout.
In async code, use ModelResourcesClient.create_async instead.
You can also create a resource from the Model resources tab under Post-Training in the console, then attach sessions to it from code.
Attach to an existing resource
To train on a resource that already exists, attach to it by ID. Use this for a resource you created in the console, one a teammate provisioned, or one you keep running across experiments.attach provisions nothing and doesn’t wait. It only confirms that the resource exists.
An attached handle doesn’t own the resource, so it never stops the resource for you. Used as a context manager, it calls
detach() on exit and leaves the resource running. Without the context manager, call detach() when you’re done. To stop the resource, call stop().Detaching on exit requires SDK version 3.2.0 or later. In earlier versions, the context manager stops the resource on exit.model_resources_id in its config.
Inspect and list resources
To check a resource’s state, callretrieve on its handle. It returns the resource’s current record, including its status and, when the status is ERROR, an error with the error code:
resource.status_details.queue_position estimates its place in line, counting the older resources of the same GPU type still waiting for GPUs, plus one. It’s unset once the resource is placed on a cluster.
To find resources without a handle, list them from the base client. Filter by status to see what’s billing right now:
list once for each status. status takes a list, but the SDK sends several values in a form the API rejects with HTTP 400.
The Model resources tab in the console shows the same resources. The Together CLI lists and inspects them with tg training ls-resources, tg training ls-sessions, tg training ls-checkpoints, and tg training get <ID>. It can’t create or stop resources, so use the SDK or the console for that.
Stop a resource
AREADY resource bills until it stops, even when nothing is training on it. It stops on its own when it reaches its idle timeout (one hour by default) without an operation, or when the billing system reports that your project’s credit limit has been exceeded or your organization is blocked. Stop it yourself as soon as you’re done:
stop returns as soon as the API accepts the stop, which is when billing ends.
The API refuses to stop a resource while any session on it is still active. Stop your sessions first, or pass force=True to stop the resource along with every session attached to it, including sessions that other processes own:
stop on the base client with the resource’s ID:
Idle timeout
A resource’s idle timeout is how long it staysREADY without activity before it stops. Becoming READY, starting a session, and every operation on any of its sessions count as activity. A resource doesn’t stop while an operation is running.
Set the timeout with inactive_timeout when you create the resource, either as an argument to ModelResourcesClient.create or as a field in the API request. The value is in seconds, from 300 (five minutes) to 86400 (24 hours), and defaults to 3600 (one hour). You can’t change it after the resource is created.
The API returns each resource with its inactive_timeout and, while it’s READY, an expires_at timestamp. expires_at is the earliest time the resource can stop for inactivity, which is its last activity plus inactive_timeout. It refreshes about once a minute and never moves earlier. The resource’s details page in the console shows the same time as Stops if idle at.
To keep a resource READY while you aren’t running operations, for example while you compute rewards, call extend_expiry on its handle. Pass seconds, from 60 to 86400, and the resource doesn’t stop for inactivity until at least that many seconds from now:
expires_at. It only moves expires_at later, so passing fewer seconds than the resource already has left changes nothing. If the resource isn’t READY or seconds is out of range, the call fails with HTTP 400, which the SDK raises as BadRequestError.
To extend the expiry of a resource you don’t have a handle for, call client.post_training.model_resources.extend_expiry("your_model_resources_id", seconds=7200) from the base client.
Manage resources in the console
To create or stop a resource without code, open the Model resources tab under Post-Training in the console. The tab lists your active resources with their status, hourly rate, and estimated spend. Your past resources appear below them. Select Create model resource to create one, or stop any resource that’s still running. Statuses match the API’s lifecycle states. Select a resource’s base model name to open its details page. While the resource isREADY, the page shows Stops if idle at: the earliest time, in your local time zone, that the resource can stop if no session runs an operation before then, followed by its idle timeout. The time comes from the resource’s expires_at, so it moves later with each operation and each extend_expiry call.
The console also shows a session’s progress and its resource’s cost while a run trains.
Next steps
Train multiple adapters
Share one resource across several sessions, so a whole sweep runs on one base model’s worth of GPUs.
Pricing
Learn how a resource bills, how to estimate a run’s cost, and what’s refunded on failure.
Post-training API reference
Look up every field for resources, sessions, and operations, with its type and default.
Supported models
Check the GPU types, training modes, and session caps for each model.