Skip to content

Releases: dstackai/dstack-enterprise

0.19.13-v1

11 Jun 10:28
@r4victor r4victor
0.19.13-v1
f087935

Choose a tag to compare

View all tags
0.19.13-v1

Clusters

Built-in InfiniBand support in dstack Docker images

The dstack default Docker images now come with built-in InfiniBand support, which includes the necessary libibverbs library and InfiniBand utilities from rdma-core. This means you can run torch distributed and other workloads utilizing NCCL, and they'll take full advantage of InfiniBand without custom Docker images.

You can try InfiniBand clusters with dstack on Nebius.

Built-in EFA support in dstack VM images

dstack switches to DLAMI as the default AWS GPU VM image from a custom one. DLAMI supports EFA out-of-the-box, so you no longer need to use a custom VM image to take advantage of EFA.

Server

GCS support for code uploads

It's now possible to configure the dstack server to use GCP Cloud Storage for code uploads. Previously, only DB and S3 storages were supported. Learn more in the Server deployment guide.

What's Changed

Full Changelog: dstackai/dstack@0.19.12...0.19.13

Contributors

olgenn, colinjc, and 2 other contributors
Loading

0.19.12-v1

04 Jun 11:18
@r4victor r4victor
0.19.12-v1
f087935

Choose a tag to compare

View all tags
0.19.12-v1

Clusters

Simplified use of MPI

startup_order and stop_criteria

New run configuration properties are introduced:

  • startup_order: any/master-first/workers-first specifies the order in which master and workers jobs are started.
  • stop_criteria: all-done/master-done specifies the criteria when a multi-node run should be considered finished.

These properties simplify running certain multi-node workloads. For example, MPI requires that workers are up and running when the master runs mpirun, so you'd use startup_order: workers-first. MPI workload can be considered done when the master is done, so you'd use stop_criteria: master-done and dstack won't wait for workers to exit.

DSTACK_MPI_HOSTFILE

dstack now automatically creates an MPI hostfile and exposes the DSTACK_MPI_HOSTFILE environment variable with the hostfile path. It can be used directly as mpirun --hostfile $DSTACK_MPI_HOSTFILE.

CLI

We've also updated how the CLI displays run and job status. Previously, the CLI displayed the internal status code which was hard to interpret. Now, the the STATUS column in dstack ps and dstack apply displays a status code which is easy to understand why run or job was terminated.

dstack ps -n 10
 NAME               BACKEND             RESOURCES                            PRICE    STATUS        SUBMITTED
 oom-task                                                                             no offers     yesterday
 oom-task           nebius (eu-north1)  cpu=2 mem=8GB disk=100GB             $0.0496  exited (127)  yesterday
 oom-task           nebius (eu-north1)  cpu=2 mem=8GB disk=100GB             $0.0496  exited (127)  yesterday
 heavy-wolverine-1                                                                    done          yesterday
   replica=0 job=0  aws (us-east-1)     cpu=4 mem=16GB disk=100GB T4:16GB:1  $0.526   exited (0)    yesterday
   replica=0 job=1  aws (us-east-1)     cpu=4 mem=16GB disk=100GB T4:16GB:1  $0.526   exited (0)    yesterday
 cursor             nebius (eu-north1)  cpu=2 mem=8GB disk=100GB             $0.0496  stopped       yesterday
 cursor             nebius (eu-north1)  cpu=2 mem=8GB disk=100GB             $0.0496  error         yesterday
 cursor             nebius (eu-north1)  cpu=2 mem=8GB disk=100GB             $0.0496  interrupted   yesterday
 cursor             nebius (eu-north1)  cpu=2 mem=8GB disk=100GB             $0.0496  aborted       yesterday

Examples

Simplified NCCL tests

With this release improvements, it became much easier to run MPI workloads with dstack. This includes NCCL tests that can now be run using the following configuration:

type: task
name: nccl-tests

nodes: 2
startup_order: workers-first
stop_criteria: master-done

image: dstackai/efa
env:
  - NCCL_DEBUG=INFO
commands:
  - cd /root/nccl-tests/build
  - |
    if [ ${DSTACK_NODE_RANK} -eq 0 ]; then
      mpirun \
        --allow-run-as-root --hostfile $DSTACK_MPI_HOSTFILE \
        -n ${DSTACK_GPUS_NUM} \
        -N ${DSTACK_GPUS_PER_NODE} \
        --mca btl_tcp_if_exclude lo,docker0 \
        --bind-to none \
        ./all_reduce_perf -b 8 -e 8G -f 2 -g 1
    else
      sleep infinity
    fi

resources:
  gpu: nvidia:4:16GB
  shm_size: 16GB

See the updated NCCL tests example for more details.

Distributed training

TRL

The new TRL example walks you through how to run distributed fine-tune using TRL, Accelerate and Deepspeed.

Axolotl

The new Axolotl example walks you through how to run distributed fine-tune using Axolotl with dstack.

What's changed

Full changelog: dstackai/dstack@0.19.11...0.19.12

Contributors

olgenn, Bihan, and 4 other contributors
Loading

0.19.11-v1

28 May 09:40
@r4victor r4victor
0.19.11-v1
f087935

Choose a tag to compare

View all tags
0.19.11-v1

Runs

Replacing conda with uv

dstack's default Docker images now come with uv installed. Installing Python packages with uv can be significantly faster than with pip or conda. Here's for example, uv vs pip times for installing torch on GCP VMs:

# time uv pip install torch
...
real    0m32.771s
user    0m29.070s
sys     0m8.300s

# time pip install torch
...
real    2m26.338s
user    1m37.514s
sys     0m16.711s

To continue supporting pip, dstack now automatically activates a virtual environment with pip available.

conda is no longer included in dstack's default Docker images. If you need to use conda, it should be installed manually:

commands:
  - wget -O miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  - bash miniconda.sh -b -p /workflow/miniconda
  - eval "$(/workflow/miniconda/bin/conda shell.bash hook)"

Plugins

Built-in rest_plugin

dstack gets support for a built-in rest_plugin that allows writing custom plugins as API servers, so you don't need to install plugins as Python packages.

Plugins implemented as API servers have advantages over plugins implemented as Python packages in some cases:

  • No dependency conflicts with dstack.
  • You can use any programming language.
  • If you run the dstack server via Docker, you don't need to extend the dstack server image with plugins or map them via volumes.

To get started, check out the plugin server example. The rest_plugin server API is documented here.

AWS

New CPU series

dstack now supports most recent AWS CPU VMs based on Intel Xeon Sapphire Rapids: M7i, C7i, and R7i. It also adds support for the burstable T3 family. Previously, only M5, C5 and t2.small CPU instances were supported.

Azure

New CPU series

dstack now supports most recent Azure CPU VMs based on Intel Xeon Sapphire Rapids: general purpose Dsv6 and memory optimized Esv6 series. Previously, only Dsv3, Esv4, and Fsv2 series were supported.

GCP

New CPU series

dstack now supports most recent GCP CPU VMs: C4, M4, H3, N4, N2. Previously, only E2 and M1 were supported.

Note that C4, M4, H3, N4 instances do not currently support Volumes since they require Hyperdisk support.

Examples

Ray+RAGEN

The new Ray+RAGEN example shows how use dstack and RAGEN to fine-tune an agent on multiple nodes.

Breaking changes

  • conda is no longer included in dstack's default Docker images.

Deprecations

  • Azure VM series Dsv3 and Esv4 are deprecated.

What's Changed

New Contributors

Full Changelog: dstackai/dstack@0.19.10...0.19.11

Contributors

Bihan, colinjc, and 4 other contributors
Loading

0.19.11rc2-v1

27 May 09:42
@r4victor r4victor
0.19.11rc2-v1
f087935

Choose a tag to compare

View all tags
0.19.11rc2-v1 Pre-release
Pre-release

What's Changed

New Contributors

Full Changelog: dstackai/dstack@0.19.10...0.19.11rc2

Contributors

Bihan, colinjc, and 4 other contributors
Loading

0.19.10-v2

22 May 10:33
@r4victor r4victor
0.19.10-v2
f087935

Choose a tag to compare

View all tags
0.19.10-v2

Linking Okta accounts

Now dstack automatically links Okta accounts to existing dstack users on first login instead of creating new users if their emails match.

[14:27:08] INFO     dstack_enterprise.services.auth.okta:70 Linked existing  
                    dstack user r4victor to Okta account pqefub12345
                    (victor@dstack.ai)
Loading

0.19.10-v1

21 May 09:31
@r4victor r4victor
0.19.10-v1
89d5de7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.19.10-v1

Runs

Priorities

Run configurations now support a new priority property that allows controlling the order in which the runs are provisioned:

type: task
nodes: 1
priority: 50
commands:
  - ...

Runs with higher priorities take precedence over runs with lower priorities. Previously, submitted jobs were processed in FIFO manner with older jobs processed first. Now, the jobs are first sorted by descending priority. Note that if a high priority run cannot be scheduled, it does not block other runs with lower priority from scheduling (a.k.a. Best effort FIFO).

The priority property is updatable, so it can be changed for already submitted runs and will take effect.

CLI

dstack project command

The new dstack project command replaces the existing dstack config command.

  1. dstack project (same as dstack project list)
$ dstack project

 PROJECT         URL                    USER            DEFAULT
 peterschmidt85  https://sky.dstack.ai  peterschmidt85
 main            http://127.0.0.1:3000  admin              ✓
  1. dstack project set-default
$ dstack project set-default peterschmidt85
OK
  1. dstack project add (similar to old dstack config, but --project is changed to --name)
$ dstack project add --name peterschmidt85 --url https://sky.dstack.ai --token 76d8dd51-0470-74a7-24ed9ec18-fb7d341
OK

dstack ps -n/--last

The dstack ps command now supports a new -n/--last parameter to show last N runs:

✗ dstack ps -n 3
 NAME             BACKEND             RESOURCES                                    PRICE    STATUS      SUBMITTED    
 good-panther-2   gcp (europe-west4)  cpu=2 mem=8GB disk=100GB                     $0.0738  terminated  49 mins ago  
 new-chipmunk-1   azure (westeurope)  cpu=2 mem=8GB disk=100GB (spot)              $0.0158  terminated  23 hours ago 
 fuzzy-panther-1  runpod (EU-RO-1)    cpu=6 mem=31GB disk=100GB RTX2000Ada:16GB:1  $0.28    terminated  yesterday

Azure

Fsv2 series

The Azure backend now supports compute-optimized Fsv2 series:

✗ dstack apply -b azure
 Project              main                           
 User                 admin                          
 Configuration        .dstack.yml                    
 Type                 dev-environment                
 Resources            cpu=4.. mem=8GB.. disk=100GB.. 
 Spot policy          auto                           
 Max price            -                              
 Retry policy         -                              
 Creation policy      reuse-or-create                
 Idle duration        5m                             
 Max duration         -                              
 Inactivity duration  -                              
 Reservation          -                              

 #  BACKEND             RESOURCES                         INSTANCE TYPE      PRICE     
 1  azure (westeurope)  cpu=4 mem=8GB disk=100GB (spot)   Standard_F4s_v2    $0.0278   
 2  azure (westeurope)  cpu=4 mem=16GB disk=100GB (spot)  Standard_D4s_v3    $0.0312   
 3  azure (westeurope)  cpu=4 mem=32GB disk=100GB (spot)  Standard_E4-2s_v4  $0.0416   
    ...                                                                                
 Shown 3 of 98 offers, $40.962max

Major bugfixes

  • [Bug]: Instances with blocks feature cannot be used for multi-node runs #2650

Deprecations

  • The dstack config CLI command is deprecated in favor of dstack project add.

What's changed

Full changelog: dstackai/dstack@0.19.9...0.19.10

Contributors

un-def, olgenn, and 4 other contributors
Loading

0.19.9-v1

15 May 09:46
@r4victor r4victor
0.19.9-v1
89d5de7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.19.9-v1

CLI

Container exit status

The CLI now displays the container exit status of each failed run or job:


Monitoring

Metrics

Previously, dstack stored and displayed only metrics within the last hour. If a run or job is finished, eventually metrics disappeared.
Now, dstack stores the last hour window of metrics for all finished runs.

AMD

On AMD, a wider range of ROCm/AMD SMI versions is now supported. Previously, for certain versions, metrics were not shown properly.

Server

Robust handling of networking issues

It sometimes happens that the dstack server cannot establish connections to running instances due to networking problems or because instances become temporarily unreachable. Previously, dstack failed jobs very quickly in such cases. Now, the server puts a graceful timeout of 2 minutes before considering jobs failed if instances are unreachable.

Runs

DSTACK_RUN_ID and DSTACK_JOB_ID

Two new environment variables are now available within runs:

  • DSTACK_RUN_ID stores the UUID of the run. It's unique for a run unlike DSTACK_RUN_NAME.
  • DSTACK_JOB_ID stores the UUID of the job submission. It's unique for every replica, job, and retry attempt.

What's Changed

New Contributors

Full Changelog: dstackai/dstack@0.19.8...0.19.9

Contributors

aaroniscode, un-def, and 4 other contributors
Loading

0.19.8-v1

07 May 15:59
@jvstme jvstme
0.19.8-v1
89d5de7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.19.8-v1

ARM

dstack now supports compute instances with ARM CPUs. To request ARM CPUs in a run or fleet configuration, specify the arm architecture in the resources.cpu property:

resources:
  cpu: arm:4..  # 4 or more ARM cores

If the hosts in an SSH fleet have ARM CPUs, dstack will automatically detect them and enable their use.

To see available offers with ARM CPUs, pass --cpu arm to the dstack offer command.

Lambda

GH200

With the lambda backend, it's now possible to use GH200 instances that come with an ARM-based 72-core NVIDIA Grace CPU and an NVIDIA H200 Tensor Core GPU, connected with a high-bandwidth, memory-coherent NVIDIA NVLink-C2C interconnect.

type: dev-environment
name: my-env

ide: vscode

resources:
  gpu: GH200:1

If Lambda has GH200 on-demand instances at the time, you'll see them when you run dstack apply:

$ dstack apply -f .dstack.yml

 #   BACKEND             RESOURCES                                      INSTANCE TYPE  PRICE
 1   lambda (us-east-3)  cpu=arm:64 mem=464GB disk=4399GB GH200:96GB:1  gpu_1x_gh200   $1.49

Note, if no GH200 is available at the moment, you can specify the retry policy in your run configuration so that dstack can run the configuration once the GPU becomes available.

Nebius

InfiniBand clusters

The nebius backend now supports InfiniBand clusters. A cluster is automatically created when you apply a fleet configuration with placement: cluster and supported GPUs: e.g. 8xH100 or 8xH200.

type: fleet
name: my-fleet

nodes: 2
placement: cluster

resources:
  gpu: H100,H200:8

A suitable InfiniBand fabric for the cluster is selected automatically. You can also limit the allowed fabrics in the backend settings.

Once the cluster is provisioned, you can benefit from its high-speed networking when running distributed tasks, such as NCCL tests or Hugging Face TRL.

Azure

Managed identities

The new vm_managed_identity backend setting allows you to configure the managed identity that is assigned to VMs created in the azure backend.

projects:
- name: main
  backends:
  - type: azure
    subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808
    tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1
    creds:
      type: default
    vm_managed_identity: dstack-rg/my-managed-identity

Make sure that dstack has the required permissions for managed identities to work.

What's changed

New contributors

Full changelog: dstackai/dstack@0.19.7...0.19.8

Contributors

un-def, r4victor, and 3 other contributors
Loading

0.19.7-v1

01 May 14:07
@un-def un-def
0.19.7-v1
89d5de7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.19.7-v1

Plugins

Run configurations have many options. While dstack aims to simplify them and provide rational defaults, teams may sometimes want to enforce their own defaults and configurations across projects.

To support this, we're introducing a plugin system that allows such enforcements to be defined programmatically. You can now define a plugin using dstack's Python SDK and bundle it with the dstack server.

For example, you can create your own plugin to override run configuration options—e.g., to prepend commands, set policies, and more.

For more information on plugin development, see the documentation and example.

Note

Plugins are currently an experimental feature. Backward compatibility is not guaranteed between releases.

Tenstorrent

The new update introduces initial support for Tenstorrent's Wormhole accelerators.

Now, if you create SSH fleets with hosts that have N150 or N300 PCIe boards, dstack will automatically detect them and allow you to use such a fleet for running dev environments, tasks, and services.

Dedicated examples for using dstack with Tenstorrent's accelerators will be published soon.

What's changed

Full changelog: dstackai/dstack@0.19.5...0.19.7

Contributors

un-def, r4victor, and 2 other contributors
Loading

0.19.5-v1

23 Apr 10:41
@r4victor r4victor
0.19.5-v1
89d5de7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.19.5-v1

CLI

Offers

You can now list available offers (hardware configurations) from the configured backends using the CLI—without needing to define a run configuration. Just run dstack offer and specify the resource requirements. The CLI will output available offers, including backend, region, instance type, resources, spot availability, and pricing:

$ dstack offer --gpu H100:1.. --max-offers 10

 #   BACKEND     REGION     INSTANCE TYPE          RESOURCES                                     SPOT  PRICE   
 1   datacrunch  FIN-01     1H100.80S.30V          30xCPU, 120GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.19   
 2   datacrunch  FIN-02     1H100.80S.30V          30xCPU, 120GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.19   
 3   datacrunch  FIN-02     1H100.80S.32V          32xCPU, 185GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.19   
 4   datacrunch  ICE-01     1H100.80S.32V          32xCPU, 185GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.19   
 5   runpod      US-KS-2    NVIDIA H100 PCIe       16xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.39   
 6   runpod      CA         NVIDIA H100 80GB HBM3  24xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.69   
 7   nebius      eu-north1  gpu-h100-sxm           16xCPU, 200GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.95   
 8   runpod      AP-JP-1    NVIDIA H100 80GB HBM3  20xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.99   
 9   runpod      CA-MTL-1   NVIDIA H100 80GB HBM3  28xCPU, 251GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.99   
 10  runpod      CA-MTL-2   NVIDIA H100 80GB HBM3  26xCPU, 125GB, 1xH100 (80GB), 100.0GB (disk)  no    $2.99   
     ...                                                                                                                
 Shown 10 of 99 offers, $127.816 max

Learn more about how the new CLI works in the reference

Configuration

Resource tags

It's now possible to set custom resource-level tags using the new tags property:

type: dev-environment
ide: vscode
tags:
  my_custom_tag: some_value
  another_tag: another_value_123

The tags property is supported by all configuration types: runs, fleets, volumes, gateways, and profiles. The tags are propagated to the underlying cloud resources on backends that support tags. Currently, it's AWS, Azure, and GCP.

Shell configuration

With the new shell property you can specify the shell used to run commands (or init for dev environments):

type: task
image: ubuntu

shell: bash
commands:
  # now we can use Bash features, e.g., arrays:
  - words=(dstack is)
  - words+=(awesome)
  - echo ${words[@]}  # prints "dstack is awesome"

GCP

A3 High and A3 Edge

dstack now automatically sets up GCP A3 High and A3 Edge instances with GPUDirect-TCPX optimized NCCL communication.

An example on how to provision an A3 High cluster and run NCCL tests on it using dstack is coming soon!

Volumes

Total cost

The UI now shows volumes total cost and termination date alongside volume price. Previously, only the price information was available.

Screenshot 2025-04-23 at 14 10 05

What's changed

Full changelog: dstackai/dstack@0.19.4...0.19.5

Contributors

un-def, Bihan, and 3 other contributors
Loading