Skip to main content
To serve a trained model, deploy one of its inference checkpoints to dedicated model inference. A low-rank adapter (LoRA) can be merged into its base model when the deployment starts, so it serves at the base model’s speed and cost. It can also be served in multi-LoRA mode on an existing deployment for quicker experiments. For production traffic, deploy the model on its own deployment rather than attaching it to a multi-LoRA deployment. Depending on the base model, you may be able to optimize performance by first preparing the adapter for FP4 inference.

Find your model’s ID

An inference checkpoint registers its weights in your project’s model registry, so you deploy it by its registry ID. Read the ID from the checkpoint’s registration:
What you deploy depends on what the checkpoint’s registration holds: To find the IDs from the terminal, run tg training ls-checkpoints --type inference --session <session_id>. The Registry artifact column holds each inference checkpoint’s registry ID and revision ID as <ID>@<REVISION_ID>. tg training get <checkpoint_id> prints the same IDs for one checkpoint. You can also look up a checkpoint’s IDs without the session.

Deploy a LoRA adapter in merged mode

To serve a LoRA adapter for production traffic, deploy it on its own deployment with --merge. Each replica merges the adapter into its base model’s weights when it starts, then serves the merged model at the base model’s speed and cost. There’s no merge job or upload. Adapters trained on models served only in FP4 can’t deploy with --merge. Prepare them for FP4 inference instead. Run tg endpoints deploy with the adapter’s ID:
Merging adds to each replica’s startup time, including replicas that autoscaling adds later. A new deployment can take 20 minutes or more to become ready. You can also deploy the adapter with the SDK.

Choose a config

To find a profile for --config, pass the adapter ID to tg models configs, which lists the deployment profiles of the adapter’s serving base model:
Pick a profile whose accelerator_type is nvidia-b200-180gb and whose adapter_mode is disabled, then pass its Config ID to --config. The profile’s accelerator_count sets the number of GPUs each replica uses. If no B200 profile is listed, contact your account team for B200 serving support.

Deploy LoRA adapters for FP4 inference

To serve an adapter on its own deployment when it was trained on a model served only in FP4, first run an FP4 preparation job. The job merges the adapter into the base model’s weights, then calibrates and quantizes the merged model to FP4. A base model that already ships in the format it serves in skips calibration and quantization. The job registers the result as a private model in your project. Estimate the job, then start it:
estimate prints the job’s price and running time without creating anything. The running time depends on the base model and excludes any wait for GPU capacity. create --watch starts the paid job and prints the prepared model’s ID when it completes. Then deploy the prepared model by that ID. For custom calibration data, job monitoring, cancellation, and the SDK, see Prepare for FP4 inference.

Deploy a merged model

Deploy a merged model, such as a full-weight checkpoint or an FP4-prepared model, with tg endpoints deploy:
deploy creates the endpoint and routes all traffic to the new deployment. The dedicated model inference quickstart walks through the rest of the flow, from choosing a config to calling the endpoint. Scale the deployment to zero or delete the endpoint when you’re done.

Serve the adapter on a multi-LoRA deployment

Instead of deploying a LoRA adapter on its own, you can attach it to a running deployment of its base model, if the base model supports multi-LoRA deployment. Attaching takes about a minute, with no extra job or upload. Each request does extra work for the adapter, and a deployment holds a limited number of adapters. Use it to compare candidates and to serve adapters with low or mixed traffic. For production traffic, deploy the adapter on its own deployment.

Next steps

Multi-LoRA deployment

Attach candidate adapters to one deployment and compare them per request.

Prepare for FP4 inference

Merge an adapter into its base model and quantize the result to FP4.

Dedicated model inference quickstart

Follow the CLI and SDK deploy flow, step by step.

Manage checkpoints

Save the inference checkpoints you deploy.