> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Configure autoscaling
> Autoscale a deployment between replica bounds, pick the right scaling metric, and stop idle deployments automatically.
Configure your deployment to scale automatically by setting limits on how many replicas it can run. You set the minimum and maximum replica count, and the platform autoscales between those bounds based on a metric that you choose.
## Replica bounds
A replica is one instance of your model running on its own hardware. Adding more replicas raises the aggregate requests per second that a deployment can serve, and also adds redundancy in case a single replica fails.
Two parameters control the number of replicas that a deployment can run:
* **`minReplicas`:** The floor. The deployment never scales below this. On a running deployment, it must be at least `1`.
* **`maxReplicas`:** The ceiling. Caps how far the deployment scales up, which determines the deployment's maximum cost.
You can set both at [creation time](/docs/dedicated-endpoints/manage#create-a-deployment) with the CLI's `tg beta endpoints deploy --min-replicas`/`--max-replicas`. To update the bounds on a running deployment, pass the deployment ID (`dep_...`) to `tg beta endpoints update`. The CLI resolves its parent endpoint automatically:
```bash CLI theme={null}
tg beta endpoints update dep_abc123 \
--min-replicas 1 \
--max-replicas 4
```
Setting `minReplicas` equal to `maxReplicas` fixes the size of the deployment and disables autoscaling. Setting both to `0` stops the deployment. A `minReplicas` of `0` paired with a positive `maxReplicas` is rejected.
An active [rollout](/docs/dedicated-endpoints/rollouts) also holds a replica floor for each deployment for its current step. If you lower `maxReplicas` below what the rollout's current step requires, the rollout pauses until you restore a compatible replica bound.
## Enable autoscaling
To enable autoscaling, set `minReplicas` to less than `maxReplicas`. With a range set but no metric specified, the platform applies a default scaling metric (concurrent in-flight requests per replica), so you can enable autoscaling without first choosing a metric or target.
To scale on a specific metric, pass `--scaling-metric` and `--scaling-target` together. A deployment scales on a single metric, so you set exactly one. Choose the metric when you first deploy:
```bash CLI theme={null}
tg beta endpoints deploy zai-org/GLM-5.2 \
--endpoint my-glm-endpoint \
--min-replicas 1 \
--max-replicas 4 \
--scaling-metric gpu_utilization \
--scaling-target 70
```
You can also set or change the metric later on a running deployment. Pass the deployment ID (`dep_...`). With the Python SDK, use snake\_case field names and set the metric target `type` explicitly:
```bash CLI theme={null}
tg beta endpoints update dep_abc123 \
--min-replicas 1 \
--max-replicas 4 \
--scaling-metric gpu_utilization \
--scaling-target 70
```
```python Python theme={null}
from together import Together
client = Together()
project_id = client.whoami().project_id
deployment = client.beta.endpoints.deployments.update(
"dep_abc123",
project_id=project_id,
endpoint_id="ep_abc123",
autoscaling={
"min_replicas": 1,
"max_replicas": 4,
"scaling_metrics": [
{
"name": "gpu_utilization",
"target": 70,
"type": "METRIC_TARGET_TYPE_UTILIZATION",
}
],
},
)
```
The SDK converts the snake\_case fields above to the API's camelCase fields. You can pass the same `autoscaling` object to `client.beta.endpoints.deployments.create`. For create requests, `model` and `config` must be fully qualified resource names, such as `projects/{project_id}/models/{model_id}` and `projects/{project_id}/configs/{config_id}`. See the [create deployment](/reference/dmi/deployments-create) and [update deployment](/reference/dmi/deployments-update) API references.
## Scaling metrics
`--scaling-metric` accepts one of nine metrics, and `--scaling-target` sets the threshold the platform holds it near. The unit of the target depends on the metric:
| Metric (`--scaling-metric`) | What `--scaling-target` sets |
| - | - |
| `inflight_requests` | Concurrent in-flight requests per replica. The default metric; the target is `8` when no metric is specified. Works for LLM engines and TTS handlers. |
| `active_sessions` | Concurrent open client WebSocket sessions per replica on a TTS deployment. Accepts `AVERAGE_VALUE` only. |
| `gpu_utilization` | GPU compute utilization, as a percentage (`0`–`100`). |
| `token_utilization` | KV-cache utilization, as a percentage (`0`–`100`). |
| `cache_hit_rate` | Prompt-cache hit rate, as a percentage (`0`–`100`). |
| `throughput_per_replica` | Tokens generated per second, per replica. |
| `ttft` | Time to first token, in milliseconds. |
| `decoding_speed` | Time per output token, in milliseconds. |
| `e2e_latency` | End-to-end request latency, in milliseconds. |
The target is interpreted in one of three ways, depending on the metric:
* **Utilization** (`gpu_utilization`, `token_utilization`, `cache_hit_rate`): an average percentage across all replicas, from `0` to `100`. `gpu_utilization` with `--scaling-target 70` holds GPU utilization near 70%.
* **Average value** (`inflight_requests`, `active_sessions`, `throughput_per_replica`): an average value across all replicas. `inflight_requests` with `--scaling-target 16` aims for about 16 concurrent requests per replica. `active_sessions` with `--scaling-target 4` aims for about four open WebSocket sessions per replica.
* **Absolute value** (`ttft`, `decoding_speed`, `e2e_latency`): a value across the whole deployment, measured at a percentile. `e2e_latency` with `--scaling-target 2000` targets a p95 of 2000 ms (2 seconds). [See below](#how-latency-is-calculated) for details.
The per-token metrics `throughput_per_replica`, `ttft`, and `decoding_speed` are recorded only for streaming responses. When scaling on one of these metrics, make sure to send [streaming requests](/docs/inference/chat/overview#stream-responses) (`"stream": true`), or the deployment won't scale correctly. (`e2e_latency` is the exception: it's measured for both streaming and non-streaming traffic.)
### Choose a metric
The nine metrics fall into three families. Which one fits depends on what you want to protect the deployment against.
* **Concurrency-driven (`inflight_requests`, `active_sessions`) is the safe default.** In-flight count is a *leading* indicator: it rises when demand outpaces service, but before latency visibly degrades. It needs no streaming and no percentile choice, and it maps directly onto how inference engines batch requests. A target of `8` says "hold each replica at about eight concurrent requests." Raise it for short-prompt chat workloads that batch well, and lower it for long-context traffic (like coding agents) that spends longer in prefill. For TTS WebSocket traffic, prefer `active_sessions` (sessions per replica). HTTP TTS traffic can keep using `inflight_requests`.
* **SLO-driven (`ttft`, `e2e_latency`, `decoding_speed`) scale on the promise you make to users.** If your contract is "first token in under a second," scaling on `ttft` targets exactly that. Latency is a *trailing* signal: by the time p95 breaches the target, users can already notice it, so pair a latency metric with honest headroom in `minReplicas` rather than letting the deployment scale from a cold floor.
* **Efficiency-driven (`gpu_utilization`, `token_utilization`, `throughput_per_replica`, `cache_hit_rate`) put cost first.** They keep replicas busy and add capacity only when the fleet is genuinely saturated. Understand what "utilized" means for your workload before you rely on them: a GPU can be busy without the workload being latency-healthy, and a utilization target near `100` leaves no headroom for arrival bursts. When you scale on one of these, watch your p95 latency in [monitoring](/docs/dedicated-endpoints/monitoring).
Use this as a quick reference when picking a metric:
| Metric (`name`) | Good to use when |
| - | - |
| `inflight_requests` | Default choice. Robust, leading, and engine-agnostic (including HTTP TTS). |
| `active_sessions` | TTS deployments with concurrent WebSocket sessions. |
| `ttft` | You have a latency SLO on responsiveness (first-token time). |
| `e2e_latency` | You have an SLO on total request-completion time. |
| `gpu_utilization` | Cost-first workloads that tolerate some latency variance. |
| `token_utilization` | Batch or throughput workloads running near engine limits. |
| `throughput_per_replica` | Sustained-generation pipelines. |
| `decoding_speed` | Guarding per-request generation speed for each user. |
| `cache_hit_rate` | Specialist: cache-heavy serving patterns. |
When the metric rises above the target, the platform raises `desiredReplicas` and new replicas [cold-start](/docs/dedicated-endpoints/concepts#cold-starts). When load falls, it scales back down after a [stabilization window](#scaling-rate-and-timing). Track progress by polling the deployment: `desiredReplicas` reflects the decision immediately, `status.scheduledReplicas` shows how many replicas the scheduler has placed, and `status.readyReplicas` catches up once cold start finishes.
When you test autoscaling, measure the change in `desiredReplicas` as scaling-decision latency and the change in `status.readyReplicas` as usable-capacity latency. Sustain the test load through the expected cold-start period. A short spike can raise `desiredReplicas` and then subside before a new replica becomes ready, so it demonstrates a scaling decision but not completed scale-up.
### How latency is calculated
To calculate the overall latency of a deployment, the platform looks at a percentile of the request latency over each measurement window and scales replicas to keep the metric near your target.
The default is the 95th percentile (`p95`), meaning that 95% of requests should come in at or below your target, allowing only the slowest 5% of requests to run longer. To track a different point in the distribution, add `--scaling-percentile` with `p50`, `p90`, `p95`, or `p99`:
```bash CLI theme={null}
tg beta endpoints update dep_abc123 \
--scaling-metric e2e_latency \
--scaling-target 2000 \
--scaling-percentile p90
```
`--scaling-percentile` applies only to the latency metrics (`ttft`, `decoding_speed`, and `e2e_latency`). The platform ignores it for other metrics.
## Scaling rate and timing
After the platform computes how many replicas a deployment needs from your metric and target, two layers shape how quickly the replica count can change.
**Stabilization windows** control how long the metric must stay above or below your target before the platform acts. Pass `--scale-up-window` and `--scale-down-window` on [`deploy`](/reference/cli/endpoints-beta#deploy) or [`update`](/reference/cli/endpoints-beta#update) to override them. When omitted, scale-up has no stabilization delay (`0` seconds), and scale-down waits five minutes before removing replicas.
**Rate limits** cap how many replicas can be added or removed per evaluation. The platform applies fleet defaults that you cannot override through the public API:
| Direction | Rate limit | Period |
| - | - | - |
| Scale up | The smaller of a 100% increase or 4 replicas | 15 seconds |
| Scale down | The smaller of a 25% decrease or 1 replica | 60 seconds |
The platform evaluates scaling roughly every 60 seconds. Each evaluation applies the stabilization windows first, then clamps the replica delta to these rate limits, then clamps the result to your [replica bounds](#replica-bounds). A deployment at zero replicas stays at zero until you raise its floor above zero.
## Automatic idle shutdown
To avoid paying for a deployment that sits idle, set an inactivity timeout. If the deployment serves no inference requests for that many minutes, the platform stops it: both [replica bounds](#replica-bounds) drop to `0`, the hardware is released, and billing stops, exactly as if you had [stopped the deployment](/docs/dedicated-endpoints/manage#stop-a-deployment) yourself.
Idle shutdown is off unless you set it. Enable it when you deploy, or on a running deployment:
```bash CLI theme={null}
# At deploy time
tg beta endpoints deploy zai-org/GLM-5.2 \
--endpoint my-glm-endpoint \
--min-replicas 1 \
--max-replicas 4 \
--inactive-timeout 60
# On a running deployment
tg beta endpoints update dep_abc123 --inactive-timeout 60
```
The timeout must be between `30` and `1440` minutes (24 hours). To turn idle shutdown off, set it to `0`. In the management API, the field is `inactiveTimeout` on the deployment (`inactive_timeout` in the Python SDK).
How the idle clock runs:
* **The clock starts when the deployment first becomes ready:** A deployment that never serves a request shuts down after its timeout elapses from the moment it first reaches `READY`, not from creation.
* **Every inference request resets it:** The deployment stays up as long as requests keep arriving.
* **Changing the timeout also resets it:** When you update the value, the new period counts from the moment of the update. Enabling or lowering the timeout on an already idle deployment grants a full fresh window rather than shutting it down immediately.
After an idle shutdown the deployment sits in `DEPLOYMENT_STATE_STOPPED` and doesn't restart on its own. To bring it back, raise both replica bounds to `1` or more. The timeout stays set, so the deployment shuts down again after another idle period.
A deployment with an inactivity timeout enabled can't take part in a [rollout](/docs/dedicated-endpoints/rollouts), as either the source or the target. Set the timeout to `0` before creating the rollout, and turn it back on after the rollout completes.
## Release hardware
A deployment runs until you stop it or an [idle shutdown](#automatic-idle-shutdown) stops it. To release the hardware yourself, stop the deployment or lower its [replica bounds](#replica-bounds) to zero (set both `minReplicas` and `maxReplicas` to `0`). A stopped deployment doesn't restart on its own. To bring it back, raise both bounds to `1` or more. See [Stop a deployment](/docs/dedicated-endpoints/manage#stop-a-deployment).
## Next steps
Query the metrics that drive autoscaling decisions.
See how replica count maps to cost.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.