Preemptible compute

Preemptible compute is in public preview for Kubernetes and Slurm clusters. There is no minimum-lifetime guarantee for preemptible nodes during preview, so design workloads that can survive losing nodes at any time.

Together GPU clusters offer two compute types:

Standard nodes (the default) are provisioned up front (synchronously) when you create or scale a cluster, and are never preempted.
Preemptible nodes fill in over time (asynchronously). You set a target, and Together provisions toward it as spare capacity becomes available. The target is not guaranteed, and preemptible nodes can be preempted at any time.

Preemptible nodes are discounted relative to on-demand at a flat rate (not a bid), and usage is metered and billed sub-hourly (every one to two minutes). See the GPU clusters overview for how compute types fit into the rest of the platform. There is no separate preemptible cluster type—you add preemptible capacity to a cluster, at create time or later. Each cluster has a standard node count and a preemptible node count, and preemptible nodes join the same cluster as the standard nodes. Every cluster requires at least one standard node (num_gpus), and you cannot convert a node between standard and preemptible in place. On Kubernetes clusters, preemptible nodes carry the label together.ai/compute-class=preemptible. Node names encode the compute type: gpu-dp is standard, gpu-preemptible-dp is preemptible. On Slurm clusters, preemptible nodes are identified by the Slurm feature preemptible. Use --constraint=preemptible in your #SBATCH directives to target them.

How preemption works

When Together reclaims a preemptible node, it drains the node for at most five minutes before removing it. Together then provisions replacement preemptible nodes toward your target as capacity becomes available, so you don’t need to re-request capacity. Here’s how preemption works for each cluster type.

Kubernetes

T+0: The node is cordoned, a TogetherPreemptionNotified Kubernetes event (type: Warning) is emitted on the node, and pods on the node receive SIGTERM.
Pods that set terminationGracePeriodSeconds (capped at 300 seconds) get up to the full window to checkpoint and exit. The node is reclaimed as soon as your pods exit, so checkpoint and exit promptly instead of sleeping through the window.
T+5:00: The node is removed, regardless of pod status. Five minutes is a hard maximum, not a guarantee that pods finish.

Slurm

T+0: Slurm drains the node. No new jobs start on it, and jobs already running keep running.
Jobs that finish within the window exit normally.
T+5:00: The node is removed, and any jobs still running on it are killed. Slurm requeues the killed jobs that were submitted with --requeue.

Checkpoint frequently. Write periodic checkpoints to shared storage (/data/home) so your job can resume from where it left off after a requeue.

Request preemptible capacity

Request a preemptible GPU target alongside the standard count, at cluster create or update, from the console, CLI, or API. See the cluster create API reference for the full schema.

Field	Request or response	Meaning
`num_preemptible_gpus`	Request (create and update)	Preemptible GPU target. Must be a multiple of 8. `project_id` is required when the target is greater than 0. Omit on update to keep the current value.
`desired_preemptible_gpus`	Response (read-only)	Echo of the target.
`allocated_preemptible_gpus`	Response (read-only)	Preemptible GPUs actually live. Rises from 0 toward the target as nodes are provisioned, and can sit below the target when capacity is tight.

Create a cluster with preemptible capacity. Preemptible capacity requires a project, so pass the global --project flag (or set TOGETHER_PROJECT_ID):

tg beta clusters create \
  --name my-cluster \
  --region us-central-2 \
  --gpu-type RTX_6000_PCI \
  --cluster-type KUBERNETES \
  --num-gpus 8 \
  --billing-type ON_DEMAND \
  --nvidia-driver-version 570 \
  --cuda-version 12.8 \
  --num-preemptible-gpus 8 \
  --project <PROJECT_ID> \
  --non-interactive --json

Scale the preemptible target on an existing cluster:

tg beta clusters update <CLUSTER_ID> --num-preemptible-gpus 16

Check the target against what is actually allocated:

tg beta clusters retrieve <CLUSTER_ID> --json \
  | jq '{status, desired_preemptible_gpus, allocated_preemptible_gpus}'

Schedule workloads on preemptible nodes

Target preemptible nodes explicitly for interruptible workers, and keep coordinators and serving replicas on standard nodes.

Kubernetes

# Interruptible worker: runs on preemptible nodes
nodeSelector:
  together.ai/compute-class: preemptible

Pin stateful anchors to standard nodes by excluding the preemptible label:

# Coordinator: never scheduled on preemptible nodes
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: together.ai/compute-class
              operator: NotIn
              values:
                - preemptible

Slurm

To run a job on preemptible nodes, add --constraint=preemptible to its #SBATCH directives. Add --requeue too, so Slurm resubmits the job if a node is reclaimed while it runs:

#!/bin/bash
#SBATCH --job-name=my-training-run
#SBATCH --output=train-%j.out
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:8
#SBATCH --constraint=preemptible
#SBATCH --requeue

CKPT_DIR=/home/$USER/checkpoints/$SLURM_JOB_NAME
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)

srun torchrun \
  --nnodes=$SLURM_JOB_NUM_NODES \
  --nproc_per_node=8 \
  --rdzv_id=$SLURM_JOB_ID \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:29500 \
  train.py \
    --checkpoint-dir $CKPT_DIR \
    --resume-from-latest

Key directives

Directive	Purpose
`--constraint=preemptible`	Schedule the job only on preemptible nodes.
`--requeue`	Resubmit the job automatically when its preemptible node is removed. The restarted job sees `SLURM_RESTART_COUNT` incremented, so your script can detect a restart and resume from its last checkpoint.
`--nodes=MIN-MAX`	Let the job start, or restart after a requeue, without waiting for every node when preemptible capacity is short (for example, `--nodes=2-4`). Slurm starts the job as soon as `MIN` nodes are free, on as many nodes in the range as are available. A running job doesn’t grow when more nodes appear later, so use a range only if your workload can resume on a different number of nodes.

Detect preemption

Kubernetes

Preemption surfaces through three channels:

In-pod (recommended): A SIGTERM handler or preStop hook reacts automatically, with no polling. See Handle preemption for examples.

Kubernetes events: Watch for TogetherPreemptionNotified events:

kubectl get events -A --field-selector reason=TogetherPreemptionNotified --watch

Together API (no kubeconfig needed): Reading a cluster returns a node_lifecycle_events array (72-hour retention, deduplicated by node and reason). Filter for reason == "TogetherPreemptionNotified". Related reasons include TogetherNodeAdded and TogetherScaledDown.
curl -s -H "Authorization: Bearer $TOGETHER_API_KEY" \ "https://api.together.ai/v1/compute/clusters/<CLUSTER_ID>" \ | jq '.node_lifecycle_events[] | select(.reason == "TogetherPreemptionNotified")'

Preemption events also appear in the console under the cluster’s Event Timeline.

Slurm

On Slurm clusters, TogetherPreemptionNotified events appear in the cluster’s node_lifecycle_events array in the Together API and in the console’s Event Timeline, the same as on Kubernetes. Inside a job, check SLURM_RESTART_COUNT at startup. Slurm sets it when it requeues a job, so a value above 0 means the job is restarting, for example after its node was reclaimed, and should resume from its latest checkpoint.

Handle preemption

Kubernetes

Pod-level: checkpoint on SIGTERM. This pattern is per-workload and requires no extra infrastructure. Claim the full grace window with terminationGracePeriodSeconds and checkpoint when the signal arrives:

apiVersion: v1
kind: Pod
metadata:
  name: training-worker
spec:
  terminationGracePeriodSeconds: 300 # claim the full 5-minute window
  nodeSelector:
    together.ai/compute-class: preemptible
  containers:
    - name: worker
      image: my-training-image
      command: ["/bin/sh", "-c"]
      args:
        - |
          trap 'echo "preempted: checkpointing"; ./checkpoint.sh; exit 0' TERM
          exec_training &
          wait
      lifecycle:
        preStop: # fires before SIGTERM reaches the container
          exec:
            command: ["/bin/sh", "-c", "./checkpoint.sh"]

The preStop hook runs first, then SIGTERM, and both count against the same grace period. Use one or the other as the checkpoint trigger, not both doing duplicate work. The preStop variant matters for containers whose main process can’t trap signals, or where PID 1 swallows them.

Cluster-level: An operator watching preemption events. For a coordinated, cluster-wide response (multi-node training, custom schedulers), run a controller that watches for TogetherPreemptionNotified events and reacts, for example by draining a job queue, triggering a coordinated checkpoint, or removing the node from a Ray or torchrun worker pool:

from kubernetes import client, config, watch

config.load_incluster_config()  # or load_kube_config() outside the cluster
v1 = client.CoreV1Api()
w = watch.Watch()

for event in w.stream(
    v1.list_event_for_all_namespaces,
    field_selector="reason=TogetherPreemptionNotified",
):
    node = event["object"].involved_object.name
    print(f"Node {node} preempted; drain window open (~5 min)")
    # React here, e.g.:
    #  - signal the training coordinator to checkpoint and rebalance
    #  - remove the node from your scheduler / worker pool
    #  - requeue pending work off the doomed node

Slurm

A job still running when its node is removed loses everything since its last checkpoint. Write checkpoints to shared storage under /home at a regular interval, and resume from the latest one when Slurm requeues the job. Checkpoint often enough that losing one interval of work is acceptable. The following job script resumes from the latest checkpoint whenever SLURM_RESTART_COUNT shows that Slurm requeued it. train.py stands in for your training script, which saves a checkpoint to --checkpoint-dir every --checkpoint-interval steps and loads the latest one when passed --resume-from-latest:

#!/bin/bash
#SBATCH --job-name=preemptible-train
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:8
#SBATCH --constraint=preemptible
#SBATCH --requeue

CKPT_DIR=/home/$USER/checkpoints/$SLURM_JOB_NAME
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)

if [ "${SLURM_RESTART_COUNT:-0}" -gt 0 ]; then
  echo "Requeued (restart $SLURM_RESTART_COUNT), resuming from the latest checkpoint"
  RESUME_FLAG="--resume-from-latest"
else
  RESUME_FLAG=""
fi

srun torchrun \
  --nnodes=$SLURM_JOB_NUM_NODES \
  --nproc_per_node=8 \
  --rdzv_id=$SLURM_JOB_ID \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:29500 \
  train.py \
    --checkpoint-dir $CKPT_DIR \
    --checkpoint-interval 500 \
    $RESUME_FLAG

When to use preemptible

Use preemptible compute for:

Checkpointed training, fine-tuning, evals, distillation, batch inference, and hyperparameter sweeps. Ray Train and PyTorch Lightning restart from checkpoints out of the box.
Bursty research: raise the target for a sweep and drop it after. The cluster stays up.

Don’t use preemptible compute for:

The only copy of a multi-day run with no checkpoints.
User-facing serving replicas with no fallback capacity.
Workloads with strict SLOs.

As a sizing reference, a checkpoint of a roughly 10B-parameter model completes in about 90 seconds, leaving comfortable headroom within the five-minute window. Checkpoint often, and treat preemptible nodes as ephemeral.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

GPU CLUSTERS

FINE-TUNING

CODE EXECUTION

ADMINISTRATION

How preemption works

Kubernetes

Slurm

Request preemptible capacity

Schedule workloads on preemptible nodes

Kubernetes

Slurm

Key directives

Detect preemption

Kubernetes

Slurm

Handle preemption

Kubernetes

Slurm

When to use preemptible

​How preemption works

​Kubernetes

​Slurm

​Request preemptible capacity

​Schedule workloads on preemptible nodes

​Kubernetes

​Slurm

​Key directives

​Detect preemption

​Kubernetes

​Slurm

​Handle preemption

​Kubernetes

​Slurm

​When to use preemptible

How preemption works

Kubernetes

Slurm

Request preemptible capacity

Schedule workloads on preemptible nodes

Kubernetes

Slurm

Key directives

Detect preemption

Kubernetes

Slurm

Handle preemption

Kubernetes

Slurm

When to use preemptible