Skip to main content
The three stages of a post-training project. Train: one model resource trains up to five adapters, shown as A, B, and C, while you pay for only one base model. Evaluate: deploy the adapters to a dedicated endpoint and compare them on live traffic, where adapter B wins. Productionize: merge and quantize the winner, B, then scale it across several replicas for production inference.
To compare several low-rank adapters (LoRAs) before you scale the best one up for production, serve them all from one dedicated model inference deployment of their shared base model, called a multi-LoRA deployment. Each request names an adapter in its model field, or leaves the adapter out to call the base model.
A multi-LoRA deployment serves only adapters from post-training inference checkpoints. You can’t attach a training checkpoint or an adapter from the fine-tuning service.

When to use a multi-LoRA deployment

Five candidate adapters, A through E, served two ways. Separate deployments give each candidate its own copy of the base model, which takes five times the GPUs, and each deployment takes several minutes to provision. One multi-LoRA deployment serves every candidate on one shared, frozen base model, which takes one deployment's GPUs, and attaching a new candidate takes under a minute.
Post-training usually produces several candidate adapters, either from a hyperparameter sweep or from inference checkpoints at different steps of a single training run. To pick a winner, you run each candidate on the same prompts and compare the outputs. Serving every candidate from one deployment makes that comparison cheaper and faster:
  • Pay for one deployment, not one per adapter: An adapter is a small set of weights on top of a frozen base model, so every candidate trained on the same base model can share one copy of that model. Five candidates on Qwen3.8 27B cost one deployment’s GPUs, not five. Attaching an adapter adds nothing to the deployment’s cost.
  • Add candidates without provisioning: A new deployment can take 20 minutes or more to provision. Attaching an adapter to a running deployment usually takes under a minute, so you can test inference checkpoints as training produces them.
  • Hold the serving environment constant: Every candidate runs on the same base model and hardware, so differences in output come from the adapters, not the serving setup. To switch candidates, change the model string.
A multi-LoRA deployment trades some speed for this flexibility. Each request does extra work for its adapter, and the first request for an adapter on each replica waits while the adapter loads. Use a multi-LoRA deployment to compare candidates, then serve the winner on its own for production inference.

How adapters load

How adapters load on a multi-LoRA deployment over time. On replica 1, an attached adapter isn't loaded until the first request for it, which waits while the replica loads the weights. Later requests run at normal speed. When the replica is full, it evicts its least recently used adapter, and the next request for that adapter waits while it loads again. Replica 2, added later by autoscaling, loads the adapter's weights independently, then serves at normal speed.
Multi-LoRA deployments load adapters dynamically. When you attach an adapter, the deployment records it but doesn’t load its weights yet. The first request that names the adapter waits while the replica downloads and loads the weights, and later requests run at normal speed. Each replica loads adapters independently, including replicas that autoscaling adds later. Each replica keeps a limited number of adapters loaded. When a request names an adapter that isn’t loaded and the replica is full, the replica evicts its least recently used adapter. The next request for the evicted adapter waits for it to load again.

Supported models

You can attach an adapter only to a deployment of the base model it was trained on, and only if that model supports multi-LoRA deployment. The checkpoints and deployment table shows which models do, and the serving base model to deploy for each. For B200 serving, use a deployment profile whose accelerator_type is nvidia-b200-180gb and whose adapter_mode is dynamic. To list the profiles of the adapter’s base model, pass your adapter ID to tg models configs. If the base model has no such profile, contact your account team for B200 serving support.

Find the adapter ID

You attach an adapter by its model registry ID (ml_...). Each LoRA inference checkpoint on a supported model registers its adapter as a model, so there’s no upload step. Only a checkpoint that registers an adapter (registration.adapter) can be attached. A checkpoint that registers a merged model instead deploys directly.
Read the ID from each of the session’s inference checkpoints:
After the session ends, list its inference checkpoints with the checkpoints API, which returns the IDs in inference_registration:

Attach an adapter

You attach an adapter to a running multi-LoRA deployment of its base model, one whose deployment profile serves adapters. With the CLI, one command creates that deployment and attaches the first adapter. In the console, you create the deployment first, then attach adapters to it.
Run tg endpoints deploy with the adapter ID, an endpoint name, and --attach-adapter. This example passes an eight-B200 GLM-5.3 profile as --config, so the adapter must be trained on GLM-5.3:
CLI
The CLI looks up the adapter’s base model, then checks the my-sweep endpoint for a deployment of that model whose profile serves adapters. If there isn’t one, it creates a one-replica deployment of the base model (and the endpoint, if it doesn’t exist yet), then attaches the adapter. On success, it confirms the attachment and prints the endpoint, with the adapter listed under its deployment.To list the base model’s profiles for --config, pass the adapter ID to tg models configs:
CLI
A new multi-LoRA deployment can take 20 minutes or more to provision. It bills per minute while its replica runs. An attached adapter’s serving state starts as ADAPTER_SERVING_STATE_LOADING and changes to ADAPTER_SERVING_STATE_READY when the adapter can serve requests. On a running deployment, it’s usually ready within a minute. To check an adapter’s state, list the attached adapters.

Send requests to an adapter

To call an adapter, set model to the endpoint string followed by a colon and the adapter ID. Send the request to https://api-inference.together.ai:
After the colon, you can use the adapter’s full model name instead of its ID. To call the base model, pass the endpoint string alone (your-project-slug/my-sweep). A request that leaves out the adapter suffix (:ml_abc123) by mistake still succeeds, but the base model answers it. Before you score an adapter’s outputs, confirm that the model field of each response echoes the full your-project-slug/my-sweep:ml_abc123 string. To call an adapter from any OpenAI-compatible client, set the client’s base URL to https://api-inference.together.ai/v1. Serverless inference can’t serve post-training adapters, so a deployment is the only way to call one. If you trained with thinking turned off, as the SFT recipes do for Qwen models, pass "chat_template_kwargs": {"enable_thinking": false} in the request (chat_template_kwargs={"enable_thinking": False} in Python) so the chat template matches training.

Attach more adapters

To compare more candidates, attach each one to the same deployment. With the CLI, run the same deploy --attach-adapter command with each adapter ID. The CLI finds the deployment it created, so you can leave out --config. If the endpoint has more than one deployment that serves adapters, pass --deployment-name to choose one. In the console, select Deploy LoRA on each checkpoint. Each deployment can hold a limited number of attached adapters, set by its deployment profile. The eight-B200 GLM-5.3 profile in the example holds 12 adapters. Check the selected profile’s capacity for other models. Once a deployment is full, attaching another adapter fails until you detach one. To make room and attach in one step, select Deploy anyway when the console reports that the deployment is at its adapter limit, or pass force=True to client.endpoints.adapters.create. Either way, the deployment detaches its oldest attached adapter first.

Compare the candidates

Score the attached candidates to pick a winner.

List attached adapters

To list each deployment’s attached adapters, run tg endpoints get my-sweep. The CLI doesn’t report their serving state yet, so check it with the SDK (version 3.2.0 or later), using the endpoint and deployment IDs from get:
Each entry has the adapter’s model ID (adapter_model_id), its attachment ID (id, which starts with dad_), its serving state (serving_state), and a load state for each cluster (per_cluster). You need the attachment ID to detach the adapter.per_cluster lists only the clusters the deployment is placed on. It’s empty while the deployment is stopped or being torn down, and for a short time after you attach the adapter.

Detach an adapter

To detach an adapter, pass its attachment ID and current etag to delete. Both come from the deployment’s list of attached adapters:
retrieve, update, and delete take the attachment ID, not the adapter’s model ID. If you detach an adapter and attach it again, it gets a new attachment ID.

Stop the deployment

A running deployment bills per replica-minute, even when no request names an adapter. When you’ve finished comparing candidates, stop it:
CLI
Stopping keeps the endpoint, the deployment, and its attached adapters. After you restart the deployment, each replica loads each adapter again on the first request that names it. To restart or delete a deployment, see Manage deployments.

Troubleshooting

  • deploy fails with ”… is a LoRA adapter. Choose how to deploy it”: Add --attach-adapter to attach the adapter to a multi-LoRA deployment.
  • deploy fails with “—attach-adapter only applies to LoRA adapter models”: The model ID isn’t a LoRA adapter in the project you’re deploying to. Pass the checkpoint’s registration.adapter.id. A registration.model.id is a merged model, which deploys without --attach-adapter.
  • deploy fails with “No fixed or dynamic LoRA config found”: The adapter’s base model doesn’t support multi-LoRA deployment. Check Supported models.
  • deploy fails with “Multiple configs found for model”: The base model has more than one profile that serves adapters, and the CLI can’t prompt you to choose one, for example when you run it with --non-interactive. Pass a dynamic config with --config.
  • deploy fails because the API rejects the config: The config isn’t in the base model’s config list, so the CLI forwarded it to the API to validate. Run tg models configs with the adapter ID, then pass a dynamic config from its output.
  • deploy fails with “Config … has adapter_mode disabled”: The config you passed to --config doesn’t serve adapters. Pass a config whose adapter_mode is dynamic.
  • deploy reports Failed Precondition or “Error attaching adapter” right after it creates a deployment: The new deployment isn’t ready to take adapters yet. It keeps running and billing, and so does the endpoint if deploy created it. To attach the adapter, wait until tg endpoints get my-sweep shows the deployment ready, then run the same deploy command again. To delete the deployment and endpoint instead, run tg endpoints rm <endpoint_id> --force, which deletes every deployment on the endpoint. If rm reports that the deployment must be stopped first, wait for it to stop, then run rm again.
  • Attaching fails right after the checkpoint is saved: Together validates each adapter before you can attach it. Wait a few minutes and retry.
  • Attaching fails because the deployment is full: Detach an adapter, select Deploy anyway in the console, or attach with force=True in the SDK.
  • Re-attaching fails with 409 right after a detach: The detach is still finishing. Wait a few seconds and retry.
  • An adapter’s serving_state is ADAPTER_SERVING_STATE_FAILED: In the adapter’s per_cluster list, read the reason and message of the entry whose state is ADAPTER_LOAD_STATE_FAILED. A reason of ENGINE_SHAPE_MISMATCH or ENGINE_INCOMPATIBLE_TARGET_MODULES means the adapter’s rank or target modules don’t fit the deployment profile.
  • retrieve, update, or delete fails with 400 and “is not a canonical adapter id”: You passed the adapter’s model ID (ml_...). Pass its attachment ID (dad_...) from the list of attached adapters.
  • Requests fail with 400 and adapter_not_registered: The ID after the colon is wrong, or the adapter isn’t attached to this deployment. List the attached adapters to check, and attach the adapter if it’s missing.
  • Requests fail with 500 and lora_load_failed: The deployment couldn’t load the adapter, and the error message says why. For example, the adapter’s LoRA rank can be incompatible with the deployment’s LoRA memory pool, even when it’s below the profile’s maximum rank. Detach the adapter.
  • Responses don’t change with the adapter: If the response’s model field has no adapter suffix, the request didn’t name an adapter, so the base model answered. Check that the model string in the request ends with a colon and the adapter ID.
  • Requests fail with “Adapter lookup failed”: Send adapter requests to https://api-inference.together.ai, not https://api.together.ai.
  • The adapter’s answer appears in reasoning_content: The base model is a thinking model. If you trained with thinking off, pass "chat_template_kwargs": {"enable_thinking": false}.

Next steps

Deploy a checkpoint

Deploy the winning adapter as a merged model for production.

Prepare for FP4 inference

Merge the winning adapter into its base model and quantize the result to FP4.

Train multiple adapters on one base model

Train the candidate adapters in one hyperparameter sweep.

Manage deployments

Scale, stop, and delete endpoints and deployments.