Skip to content

Scaling & operations

In a fleet, models are deployed onto workers and scheduled by the control plane (VRAM-fit placement, auto-start, idle-stop).

Terminal window
atlas deploy <model> # place a model instance on the fleet
atlas scale <model> --replicas N # scale a deployed model
atlas stop <model> # stop a deployment
atlas deployments # list deployments

See the CLI reference for worker management (atlas workers list, atlas workers remove).

Workers are stateless executors that self-register and are removed on heartbeat timeout:

  • Graceful drain on SIGTERM — a worker stops accepting new requests and lets its in-flight requests finish before exit, so ASG scale-in and spot interruptions don’t drop live requests.
  • Spot instances work naturally — an interrupted worker is just a heartbeat timeout; the scheduler re-places its model instances on remaining capacity.
  • Heartbeat-timeout removal — vanished workers leave the pool automatically.

Each worker caches weights on local NVMe or EBS, so a model is pulled once per worker and subsequent cold boots are fast. Weights are sourced from Hugging Face or an https:// GGUF URL (via the model catalog); pin them to a fast local disk on each worker so you don’t re-pull the same 140GB on every restart.

v1 scaling is “you set ASG desired capacity.” Atlas exposes a queue-depth / utilization signal on /metrics that an external autoscaler (e.g. an ASG policy) can consume, rather than Atlas calling cloud APIs itself.