> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-LoRA deployment

> Attach adapters from post-training inference checkpoints to one dedicated deployment of their base model, and choose an adapter per request.

![The three stages of a post-training project. Train: one model resource trains up to five adapters, shown as A, B, and C, while you pay for only one base model. Evaluate: deploy the adapters to a dedicated endpoint and compare them on live traffic, where adapter B wins. Productionize: merge and quantize the winner, B, then scale it across several replicas for production inference.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/train-evaluate-productionize.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=5bad9402d75c716be9b8945752c0cf7a)

To compare several low-rank adapters (LoRAs) before you scale the best one up for production, serve them all from one [dedicated model inference](https://docs.together.ai/docs/dedicated-endpoints/overview.md) deployment of their shared base model, called a multi-LoRA deployment. Each request names an adapter in its `model` field, or leaves the adapter out to call the base model.

> [!NOTE]
>
> A multi-LoRA deployment serves only adapters from post-training inference checkpoints. You can't attach a training checkpoint or an adapter from the [fine-tuning service](https://docs.together.ai/docs/fine-tuning/overview.md).

## When to use a multi-LoRA deployment

![Five candidate adapters, A through E, served two ways. Separate deployments give each candidate its own copy of the base model, which takes five times the GPUs, and each deployment takes several minutes to provision. One multi-LoRA deployment serves every candidate on one shared, frozen base model, which takes one deployment's GPUs, and attaching a new candidate takes under a minute.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/separate-vs-multi-lora-deployments.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=c4264cea7b237e29a48bef796d59c5ec)

Post-training usually produces several candidate adapters, either from a [hyperparameter sweep](https://docs.together.ai/docs/post-training/multi-lora.md) or from inference checkpoints at different steps of a single training run. To pick a winner, you run each candidate on the same prompts and compare the outputs. Serving every candidate from one deployment makes that comparison cheaper and faster:

* **Pay for one deployment, not one per adapter:** An adapter is a small set of weights on top of a frozen base model, so every candidate trained on the same base model can share one copy of that model. Five candidates on Qwen3.8 27B cost one deployment's GPUs, not five. Attaching an adapter adds nothing to the deployment's cost.
* **Add candidates without provisioning:** A new deployment can take 20 minutes or more to provision. Attaching an adapter to a running deployment usually takes under a minute, so you can test inference checkpoints as training produces them.
* **Hold the serving environment constant:** Every candidate runs on the same base model and hardware, so differences in output come from the adapters, not the serving setup. To switch candidates, change the `model` string.

A multi-LoRA deployment trades some speed for this flexibility. Each request does extra work for its adapter, and the first request for an adapter on each replica waits while the [adapter loads](#how-adapters-load). Use a multi-LoRA deployment to compare candidates, then [serve the winner on its own](https://docs.together.ai/docs/post-training/deployment.md#deploy-a-lora-adapter-in-merged-mode) for production inference.

## How adapters load

![How adapters load on a multi-LoRA deployment over time. On replica 1, an attached adapter isn't loaded until the first request for it, which waits while the replica loads the weights. Later requests run at normal speed. When the replica is full, it evicts its least recently used adapter, and the next request for that adapter waits while it loads again. Replica 2, added later by autoscaling, loads the adapter's weights independently, then serves at normal speed.](https://mintcdn.com/togetherai-52386018/Ta7hn_4Cp_DKcb3y/images/post-training/adapter-loading-timeline.png?fit=max\&auto=format\&n=Ta7hn_4Cp_DKcb3y\&q=85\&s=5349e79b39a7d2a0b6f9d173560304a5)

Multi-LoRA deployments load adapters dynamically. When you attach an adapter, the deployment records it but doesn't load its weights yet. The first request that names the adapter waits while the replica downloads and loads the weights, and later requests run at normal speed. Each replica loads adapters independently, including replicas that autoscaling adds later.

Each replica keeps a limited number of adapters loaded. When a request names an adapter that isn't loaded and the replica is full, the replica evicts its least recently used adapter. The next request for the evicted adapter waits for it to load again.

## Supported models

You can attach an adapter only to a deployment of the base model it was trained on, and only if that model supports multi-LoRA deployment. The [checkpoints and deployment table](https://docs.together.ai/docs/post-training/supported-models.md#checkpoints-and-deployment) shows which models do, and the serving base model to deploy for each.

For B200 serving, use a deployment profile whose `accelerator_type` is `nvidia-b200-180gb` and whose `adapter_mode` is `dynamic`. To list the profiles of the adapter's base model, pass your [adapter ID](#find-the-adapter-id) to `tg models configs`. If the base model has no such profile, contact your account team for B200 serving support.

## Find the adapter ID

You attach an adapter by its model registry ID (`ml_...`). Each LoRA [inference checkpoint](https://docs.together.ai/docs/post-training/checkpoints.md#inference-checkpoints) on a supported model registers its adapter as a model, so there's no upload step. Only a checkpoint that registers an adapter (`registration.adapter`) can be attached. A checkpoint that registers a merged model instead [deploys directly](https://docs.together.ai/docs/post-training/deployment.md#deploy-a-merged-model).

> [tab] SDK
>
> Read the ID from each of the session's inference checkpoints:
>
> ```python theme={null}
> from together import Together
>
> client = Together()
> session = client.post_training.sessions.retrieve("<session_id>")
> for checkpoint in session.inference_checkpoints:
>     if checkpoint.registration and checkpoint.registration.adapter:
>         print(checkpoint.step, checkpoint.registration.adapter.id)
> ```
>
> After the session ends, list its inference checkpoints with the checkpoints API, which returns the IDs in `inference_registration`:
>
> ```python theme={null}
> page = client.post_training.checkpoints.list(
>     session_id="<session_id>",
>     type="CHECKPOINT_TYPE_INFERENCE",
> )
> for checkpoint in page.data:
>     if (
>         checkpoint.inference_registration
>         and checkpoint.inference_registration.adapter
>     ):
>         print(checkpoint.step, checkpoint.inference_registration.adapter.id)
> ```

> [tab] Console
>
> [Training sessions](https://api.together.ai/post-training/sessions)
>
> Open the session, then copy the adapter ID from the inference checkpoint's row.

## Attach an adapter

You attach an adapter to a running multi-LoRA deployment of its base model, one whose deployment profile serves adapters. With the CLI, one command creates that deployment and attaches the first adapter. In the console, you create the deployment first, then attach adapters to it.

> [tab] CLI
>
> Run `tg endpoints deploy` with the adapter ID, an endpoint name, and `--attach-adapter`. This example passes an eight-B200 GLM-5.3 profile as `--config`, so the adapter must be trained on GLM-5.3:
>
> ```bash CLI theme={null}
> tg endpoints deploy ml_abc123 \
>   --endpoint my-sweep \
>   --attach-adapter \
>   --config cr_CgTLn7NQ61z5Uv9AEUWL2
> ```
>
> The CLI looks up the adapter's base model, then checks the `my-sweep` endpoint for a deployment of that model whose profile serves adapters. If there isn't one, it creates a one-replica deployment of the base model (and the endpoint, if it doesn't exist yet), then attaches the adapter. On success, it confirms the attachment and prints the endpoint, with the adapter listed under its deployment.
>
> | Parameter | Description |
> | - | - |
> | `ml_abc123` (positional) | The adapter's [registry ID](#find-the-adapter-id) (`ml_...`). |
> | `--endpoint` | The endpoint to attach to. The CLI creates it if it doesn't exist. |
> | `--attach-adapter` | Attaches the adapter to a multi-LoRA deployment of its base model on the endpoint, and creates that deployment if there isn't one. |
> | `--config` | The profile for a deployment the CLI creates. Pass one of the base model's [`dynamic` profiles](#supported-models). Optional when attaching to an existing deployment. |
> | `--deployment-name` | Optional. Attaches to the endpoint's deployment of this name, or names the deployment the CLI creates. |
> | [Replica, autoscaling, placement, and `--inactive-timeout` flags](https://docs.together.ai/reference/cli/endpoints#deploy) | Optional. Configure a deployment the CLI creates. When the CLI attaches to an existing deployment, it ignores them and prints a warning. |
>
> To list the base model's profiles for `--config`, pass the adapter ID to `tg models configs`:
>
> ```bash CLI theme={null}
> tg models configs ml_abc123
> ```

> [tab] Console
>
> [Endpoints](https://api.together.ai/endpoints)
>
> **1. Create a multi-LoRA deployment**
>
> On the **Endpoints** page, select **New endpoint**. Enter an **Endpoint name** and a **Deployment name**, then under **Model**, select your adapters' [serving base model](https://docs.together.ai/docs/post-training/supported-models.md#checkpoints-and-deployment), for example GLM-5.3 FP8.
>
> **2. Turn on LoRA adapters**
>
> Turn on **Enable LoRA adapters**. With adapters turned on, **Deployment profiles** lists only the profiles that serve adapters. Select a B200 card, then select **Create endpoint**.
>
> **3. Wait for the deployment**
>
> Open the endpoint, then open the deployment from the endpoint's **Overview** tab. Wait until the **Status** card shows **Ready**. You can't attach an adapter while the deployment is still provisioning.
>
> **4. Open the Deploy LoRA dialog**
>
> On the [Training sessions](https://api.together.ai/post-training/sessions) page, open the session, then select **Deploy LoRA** from the inference checkpoint's actions menu.
>
> **5. Choose the deployment**
>
> **Where to deploy** lists the ready deployments of the adapter's base model that have LoRA adapters turned on. Select your deployment, then select **Next**.
>
> A deployment that already serves this adapter is marked **Already deployed**. If you select it and select **Next**, the dialog offers **View deployment** instead of **Deploy**.
>
> **6. Deploy**
>
> Review the adapter and the deployment, then select **Deploy**. The dialog shows **Attaching adapter** while the deployment loads the adapter, then **Adapter attached** and the `model` string for calling it.
>
> If you open **Deploy LoRA** before you have a ready deployment, the dialog shows a **Create one** link, which opens the create form with the base model and **Enable LoRA adapters** already set.

A new multi-LoRA deployment can take 20 minutes or more to provision. It bills per minute while its replica runs.

An attached adapter's serving state starts as `ADAPTER_SERVING_STATE_LOADING` and changes to `ADAPTER_SERVING_STATE_READY` when the adapter can serve requests. On a running deployment, it's usually ready within a minute. To check an adapter's state, [list the attached adapters](#list-attached-adapters).

## Send requests to an adapter

To call an adapter, set `model` to the endpoint string followed by a colon and the adapter ID. Send the request to `https://api-inference.together.ai`:

```bash cURL theme={null}
curl -s -X POST https://api-inference.together.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-project-slug/my-sweep:ml_abc123",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 128
  }'
```

```python Python theme={null}
from together import Together

client = Together(base_url="https://api-inference.together.ai")

response = client.chat.completions.create(
    model="your-project-slug/my-sweep:ml_abc123",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=128,
)
print(response.model)
print(response.choices[0].message.content)
```

After the colon, you can use the adapter's full model name instead of its ID. To call the base model, pass the endpoint string alone (`your-project-slug/my-sweep`).

A request that leaves out the adapter suffix (`:ml_abc123`) by mistake still succeeds, but the base model answers it. Before you score an adapter's outputs, confirm that the `model` field of each response echoes the full `your-project-slug/my-sweep:ml_abc123` string.

To call an adapter from any OpenAI-compatible client, set the client's base URL to `https://api-inference.together.ai/v1`. Serverless inference can't serve post-training adapters, so a deployment is the only way to call one.

If you trained with thinking turned off, as the [SFT recipes](https://docs.together.ai/docs/post-training/sft-recipes.md) do for Qwen models, pass `"chat_template_kwargs": {"enable_thinking": false}` in the request (`chat_template_kwargs={"enable_thinking": False}` in Python) so the chat template matches training.

## Attach more adapters

To compare more candidates, attach each one to the same deployment. With the CLI, run the same `deploy --attach-adapter` command with each adapter ID. The CLI finds the deployment it created, so you can leave out `--config`. If the endpoint has more than one deployment that serves adapters, pass `--deployment-name` to choose one. In the console, select **Deploy LoRA** on each checkpoint.

Each deployment can hold a limited number of attached adapters, set by its deployment profile. The eight-B200 GLM-5.3 profile in the example holds 12 adapters. Check the selected profile's capacity for other models. Once a deployment is full, attaching another adapter fails until you [detach one](#detach-an-adapter). To make room and attach in one step, select **Deploy anyway** when the console reports that the deployment is at its adapter limit, or pass `force=True` to `client.endpoints.adapters.create`. Either way, the deployment detaches its oldest attached adapter first.

## Compare the candidates

[Score the attached candidates](https://docs.together.ai/docs/post-training/evaluate.md#compare-candidates) to pick a winner.

## List attached adapters

> [tab] CLI
>
> To list each deployment's attached adapters, run `tg endpoints get my-sweep`. The CLI doesn't report their serving state yet, so check it with the SDK (version 3.2.0 or later), using the endpoint and deployment IDs from `get`:
>
> ```python theme={null}
> from together import Together
>
> client = Together()
> identity = client.whoami()
> if identity.type != "project_api_key":
>     raise RuntimeError("TOGETHER_API_KEY must be a project API key")
> project_id = identity.project.id
>
> for adapter in client.endpoints.adapters.list(
>     "ep_abc123",
>     "dep_abc123",
>     project_id=project_id,
> ):
>     print(adapter.adapter_model_id, adapter.id, adapter.serving_state)
> ```
>
> ```text theme={null}
> ml_abc123 dad_abc123 ADAPTER_SERVING_STATE_READY
> ```
>
> Each entry has the adapter's model ID (`adapter_model_id`), its attachment ID (`id`, which starts with `dad_`), its serving state (`serving_state`), and a load state for each cluster (`per_cluster`). You need the attachment ID to [detach the adapter](#detach-an-adapter).
>
> `per_cluster` lists only the clusters the deployment is placed on. It's empty while the deployment is stopped or being torn down, and for a short time after you attach the adapter.

> [tab] Console
>
> [Endpoints](https://api.together.ai/endpoints)
>
> Open the endpoint, then open the deployment. The **LoRA adapters** card lists each attached adapter with its **Status**, which is **Loading**, **Ready**, or **Failed**.

## Detach an adapter

To detach an adapter, pass its attachment ID and current `etag` to `delete`. Both come from the deployment's list of attached adapters:

```python theme={null}
from together import Together

client = Together()
identity = client.whoami()
if identity.type != "project_api_key":
    raise RuntimeError("TOGETHER_API_KEY must be a project API key")
project_id = identity.project.id

for adapter in client.endpoints.adapters.list(
    "ep_abc123",
    "dep_abc123",
    project_id=project_id,
):
    if adapter.adapter_model_id == "ml_abc123":
        client.endpoints.adapters.delete(
            adapter.id,
            project_id=project_id,
            endpoint_id="ep_abc123",
            deployment_id="dep_abc123",
            etag=adapter.etag,
        )
        break
```

`retrieve`, `update`, and `delete` take the attachment ID, not the adapter's model ID. If you detach an adapter and attach it again, it gets a new attachment ID.

## Stop the deployment

A running deployment bills per replica-minute, even when no request names an adapter. When you've finished comparing candidates, stop it:

> [tab] CLI
>
> ```bash CLI theme={null}
> tg endpoints update dep_abc123 --min-replicas 0 --max-replicas 0
> ```

> [tab] Console
>
> [Endpoints](https://api.together.ai/endpoints)
>
> On the deployment's page, select **Stop deployment**.

Stopping keeps the endpoint, the deployment, and its attached adapters. After you restart the deployment, each replica loads each adapter again on the first request that names it. To restart or delete a deployment, see [Manage deployments](https://docs.together.ai/docs/dedicated-endpoints/manage.md#stop-a-deployment).

## Troubleshooting

* **`deploy` fails with "... is a LoRA adapter. Choose how to deploy it":** Add `--attach-adapter` to attach the adapter to a multi-LoRA deployment.
* **`deploy` fails with "--attach-adapter only applies to LoRA adapter models":** The model ID isn't a LoRA adapter in the project you're deploying to. Pass the checkpoint's `registration.adapter.id`. A `registration.model.id` is a merged model, which [deploys without `--attach-adapter`](https://docs.together.ai/docs/post-training/deployment#deploy-a-merged-model).
* **`deploy` fails with "No fixed or dynamic LoRA config found":** The adapter's base model doesn't support multi-LoRA deployment. Check [Supported models](#supported-models).
* **`deploy` fails with "Multiple configs found for model":** The base model has more than one profile that serves adapters, and the CLI can't prompt you to choose one, for example when you run it with `--non-interactive`. Pass a `dynamic` config with `--config`.
* **`deploy` fails because the API rejects the config:** The config isn't in the base model's config list, so the CLI forwarded it to the API to validate. Run `tg models configs` with the adapter ID, then pass a `dynamic` config from its output.
* **`deploy` fails with "Config ... has adapter\_mode disabled":** The config you passed to `--config` doesn't serve adapters. Pass a config whose `adapter_mode` is `dynamic`.
* **`deploy` reports `Failed Precondition` or "Error attaching adapter" right after it creates a deployment:** The new deployment isn't ready to take adapters yet. It keeps running and billing, and so does the endpoint if `deploy` created it. To attach the adapter, wait until `tg endpoints get my-sweep` shows the deployment ready, then run the same `deploy` command again. To delete the deployment and endpoint instead, run `tg endpoints rm <endpoint_id> --force`, which deletes every deployment on the endpoint. If `rm` reports that the deployment must be stopped first, wait for it to stop, then run `rm` again.
* **Attaching fails right after the checkpoint is saved:** Together validates each adapter before you can attach it. Wait a few minutes and retry.
* **Attaching fails because the deployment is full:** [Detach an adapter](#detach-an-adapter), select **Deploy anyway** in the console, or attach with `force=True` in the SDK.
* **Re-attaching fails with `409` right after a detach:** The detach is still finishing. Wait a few seconds and retry.
* **An adapter's `serving_state` is `ADAPTER_SERVING_STATE_FAILED`:** In the adapter's `per_cluster` list, read the `reason` and `message` of the entry whose `state` is `ADAPTER_LOAD_STATE_FAILED`. A `reason` of `ENGINE_SHAPE_MISMATCH` or `ENGINE_INCOMPATIBLE_TARGET_MODULES` means the adapter's rank or target modules don't fit the deployment profile.
* **`retrieve`, `update`, or `delete` fails with `400` and "is not a canonical adapter id":** You passed the adapter's model ID (`ml_...`). Pass its attachment ID (`dad_...`) from the [list of attached adapters](#list-attached-adapters).
* **Requests fail with `400` and `adapter_not_registered`:** The ID after the colon is wrong, or the adapter isn't attached to this deployment. [List the attached adapters](#list-attached-adapters) to check, and attach the adapter if it's missing.
* **Requests fail with `500` and `lora_load_failed`:** The deployment couldn't load the adapter, and the error message says why. For example, the adapter's LoRA rank can be incompatible with the deployment's LoRA memory pool, even when it's below the profile's maximum rank. [Detach the adapter](#detach-an-adapter).
* **Responses don't change with the adapter:** If the response's `model` field has no adapter suffix, the request didn't name an adapter, so the base model answered. Check that the `model` string in the request ends with a colon and the adapter ID.
* **Requests fail with "Adapter lookup failed":** Send adapter requests to `https://api-inference.together.ai`, not `https://api.together.ai`.
* **The adapter's answer appears in `reasoning_content`:** The base model is a thinking model. If you trained with thinking off, pass `"chat_template_kwargs": {"enable_thinking": false}`.

## Next steps

> [card] [Deploy a checkpoint](https://docs.together.ai/docs/post-training/deployment.md)
>
> Deploy the winning adapter as a merged model for production.

> [card] [Prepare for FP4 inference](https://docs.together.ai/docs/post-training/prepare-for-fp4-inference.md)
>
> Merge the winning adapter into its base model and quantize the result to FP4.

> [card] [Train multiple adapters on one base model](https://docs.together.ai/docs/post-training/multi-lora.md)
>
> Train the candidate adapters in one hyperparameter sweep.

> [card] [Manage deployments](https://docs.together.ai/docs/dedicated-endpoints/manage.md)
>
> Scale, stop, and delete endpoints and deployments.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.