diff --git a/docs/docs/concepts/gateways.md b/docs/docs/concepts/gateways.md index 55573bd747..4908ca8820 100644 --- a/docs/docs/concepts/gateways.md +++ b/docs/docs/concepts/gateways.md @@ -32,8 +32,6 @@ domain: example.com -A domain name is required to create a gateway. - To create or update the gateway, simply call the [`dstack apply`](../reference/cli/dstack/apply.md) command:
@@ -53,6 +51,12 @@ Provisioning... ## Configuration options +### Domain + +A gateway requires a `domain` to be specified in the configuration before creation. The domain is used to generate service endpoints (e.g. `.`). + +Once the gateway is created and assigned a hostname, configure your DNS by adding a wildcard record for `*.` (e.g. `*.example.com`). The record should point to the gateway's hostname and should be of type `A` if the hostname is an IP address (most cases), or of type `CNAME` if the hostname is another domain (some private gateways and Kubernetes). + ### Backend You can create gateways with the `aws`, `azure`, `gcp`, or `kubernetes` backends, but that does not limit where services run. A gateway can use one backend while services run on any other backend supported by dstack, including backends where gateways themselves cannot be created. @@ -61,27 +65,6 @@ You can create gateways with the `aws`, `azure`, `gcp`, or `kubernetes` backends Gateways in `kubernetes` backend require an external load balancer. Managed Kubernetes solutions usually include a load balancer. For self-hosted Kubernetes, you must provide a load balancer by yourself. -### Instance type - -By default, `dstack` provisions a small, low-cost instance for the gateway. If you expect to run high-traffic services, you can configure a larger instance type using the `instance_type` property. - -
- -```yaml -type: gateway -name: example-gateway - -backend: aws -region: eu-west-1 - -# (Optional) Override the gateway instance type -instance_type: t3.large - -domain: example.com -``` - -
- ### Router By default, the gateway uses its own load balancer to route traffic between replicas. However, you can delegate this responsibility to a specific router by setting the `router` property. Currently, the only supported external router is `sglang`. @@ -123,21 +106,65 @@ router: > > Support for prefill/decode disaggregation and auto-scaling based on inter-token latency is coming soon. +### Certificate + +By default, when you run a service with a gateway, `dstack` provisions an SSL certificate via Let's Encrypt for the configured domain. This automatically enables HTTPS for the service endpoint. + +If you disable [public IP](#public-ip) (e.g. to make the gateway private) or if you simply don't need HTTPS, you can set `certificate` to `null`. + +> Note, by default services set [`https`](../reference/dstack.yml/service.md#https) to `true` which requires a certificate. You can set `https` to `auto` to detect if the gateway supports HTTPS or not automatically. + +??? info "Certificate types" + `dstack` supports the following certificate types: + + * `lets-encrypt` (default) — Automatic certificates via [Let's Encrypt](https://letsencrypt.org/). Requires a [public IP](#public-ip). + * `acm` — Certificates managed by [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/). AWS-only. TLS is terminated at the load balancer, not at the gateway. + * `null` — No certificate. Services will use HTTP. + ### Public IP -If you don't need/want a public IP for the gateway, you can set the `public_ip` to `false` (the default value is `true`), making the gateway private. +If you don't need a public IP for the gateway, you can set `public_ip` to `false` (the default is `true`), making the gateway private. + Private gateways are currently supported in `aws` and `gcp` backends. -!!! info "Reference" - For all gateway configuration options, refer to the [reference](../reference/dstack.yml/gateway.md). +
-## Update DNS records +```yaml +type: gateway +name: private-gateway + +backend: aws +region: eu-west-1 +domain: example.com + +public_ip: false +certificate: null +``` + +
+ +### Instance type + +By default, `dstack` provisions a small, low-cost instance for the gateway. If you expect to run high-traffic services, you can configure a larger instance type using the `instance_type` property. + +
-Once the gateway is assigned a hostname, go to your domain's DNS settings -and add a DNS record for `*.`, e.g. `*.example.com`. -The record should point to the gateway's hostname shown in `dstack` -and should be of type `A` if the hostname is an IP address (most cases), -or of type `CNAME` if the hostname is another domain (some private gateways and Kubernetes). +```yaml +type: gateway +name: example-gateway + +backend: aws +region: eu-west-1 + +instance_type: t3.large + +domain: example.com +``` + +
+ +!!! info "Reference" + For all gateway configuration options, refer to the [reference](../reference/dstack.yml/gateway.md). ## Manage gateways diff --git a/docs/docs/reference/dstack.yml/gateway.md b/docs/docs/reference/dstack.yml/gateway.md index 1d74c95705..33fbeb4190 100644 --- a/docs/docs/reference/dstack.yml/gateway.md +++ b/docs/docs/reference/dstack.yml/gateway.md @@ -22,6 +22,8 @@ The `gateway` configuration type allows creating and updating [gateways](../../c ### `certificate` +Set to `null` to disable certificates (e.g. for [private gateways](../../concepts/gateways.md#public-ip)). + === "Let's encrypt" #SCHEMA# dstack._internal.core.models.gateways.LetsEncryptGatewayCertificate diff --git a/src/dstack/_internal/core/models/configurations.py b/src/dstack/_internal/core/models/configurations.py index 345f082f3d..87e38b496a 100644 --- a/src/dstack/_internal/core/models/configurations.py +++ b/src/dstack/_internal/core/models/configurations.py @@ -856,9 +856,13 @@ class ServiceConfigurationParams(CoreModel): ) ), ] = None - https: Annotated[bool, Field(description="Enable HTTPS if running with a gateway")] = ( - SERVICE_HTTPS_DEFAULT - ) + https: Annotated[ + Union[bool, Literal["auto"]], + Field( + description="Enable HTTPS if running with a gateway." + " Set to `auto` to determine automatically based on gateway configuration" + ), + ] = SERVICE_HTTPS_DEFAULT auth: Annotated[bool, Field(description="Enable the authorization")] = True scaling: Annotated[ diff --git a/src/dstack/_internal/core/models/gateways.py b/src/dstack/_internal/core/models/gateways.py index 816395fc82..7f09d3df18 100644 --- a/src/dstack/_internal/core/models/gateways.py +++ b/src/dstack/_internal/core/models/gateways.py @@ -77,7 +77,10 @@ class GatewayConfiguration(CoreModel): public_ip: Annotated[bool, Field(description="Allocate public IP for the gateway")] = True certificate: Annotated[ Optional[AnyGatewayCertificate], - Field(description="The SSL certificate configuration. Defaults to `type: lets-encrypt`"), + Field( + description="The SSL certificate configuration." + " Set to `null` to disable. Defaults to `type: lets-encrypt`" + ), ] = LetsEncryptGatewayCertificate() tags: Annotated[ Optional[Dict[str, str]], diff --git a/src/dstack/_internal/server/services/services/__init__.py b/src/dstack/_internal/server/services/services/__init__.py index 8dba43ea85..8f0849050f 100644 --- a/src/dstack/_internal/server/services/services/__init__.py +++ b/src/dstack/_internal/server/services/services/__init__.py @@ -22,7 +22,6 @@ ) from dstack._internal.core.models.configurations import ( DEFAULT_REPLICA_GROUP_NAME, - SERVICE_HTTPS_DEFAULT, ServiceConfiguration, ) from dstack._internal.core.models.gateways import GatewayConfiguration, GatewayStatus @@ -241,7 +240,7 @@ def _register_service_in_server(run_model: RunModel, run_spec: RunSpec) -> Servi "Service with SGLang router configuration requires a gateway. " "Please configure a gateway with the SGLang router enabled." ) - if run_spec.configuration.https != SERVICE_HTTPS_DEFAULT: + if run_spec.configuration.https is False: # Note: if the user sets `https: `, it will be ignored silently # TODO: in 0.19, make `https` Optional to be able to tell if it was set or omitted raise ServerClientError( @@ -416,7 +415,14 @@ async def unregister_replica(session: AsyncSession, job_model: JobModel): def _get_service_https(run_spec: RunSpec, configuration: GatewayConfiguration) -> bool: assert run_spec.configuration.type == "service" - if not run_spec.configuration.https: + https = run_spec.configuration.https + if https == "auto": + if configuration.certificate is None: + return False + if configuration.certificate.type == "acm": + return False + return True + if not https: return False if configuration.certificate is not None and configuration.certificate.type == "acm": return False diff --git a/src/tests/_internal/server/services/services/test_services.py b/src/tests/_internal/server/services/services/test_services.py new file mode 100644 index 0000000000..1ee3bf70e4 --- /dev/null +++ b/src/tests/_internal/server/services/services/test_services.py @@ -0,0 +1,100 @@ +from typing import Literal, Optional, Union +from unittest.mock import MagicMock + +import pytest + +from dstack._internal.core.errors import ServerClientError +from dstack._internal.core.models.backends.base import BackendType +from dstack._internal.core.models.configurations import ServiceConfiguration +from dstack._internal.core.models.gateways import ( + ACMGatewayCertificate, + AnyGatewayCertificate, + GatewayConfiguration, + LetsEncryptGatewayCertificate, +) +from dstack._internal.core.models.runs import RunSpec +from dstack._internal.server.services.services import ( + _get_service_https, + _register_service_in_server, +) +from dstack._internal.server.testing.common import get_run_spec + + +def _service_run_spec(https: Union[bool, Literal["auto"]] = "auto") -> RunSpec: + return get_run_spec( + repo_id="test-repo", + configuration=ServiceConfiguration(commands=["python serve.py"], port=8000, https=https), + ) + + +def _gateway_config( + certificate: Optional[AnyGatewayCertificate] = LetsEncryptGatewayCertificate(), +) -> GatewayConfiguration: + return GatewayConfiguration( + backend=BackendType.AWS, + region="us-east-1", + certificate=certificate, + ) + + +def _mock_run_model() -> MagicMock: + run_model = MagicMock() + run_model.project.name = "test-project" + run_model.run_name = "test-run" + return run_model + + +class TestServiceConfigurationHttps: + def test_default_is_true(self) -> None: + conf = ServiceConfiguration(commands=["python serve.py"], port=8000) + assert conf.https is True + + def test_accepts_auto(self) -> None: + conf = ServiceConfiguration(commands=["python serve.py"], port=8000, https="auto") + assert conf.https == "auto" + + +class TestGetServiceHttps: + def test_auto_resolves_to_true_with_lets_encrypt_gateway(self) -> None: + run_spec = _service_run_spec(https="auto") + gw = _gateway_config(certificate=LetsEncryptGatewayCertificate()) + assert _get_service_https(run_spec, gw) is True + + def test_auto_resolves_to_false_when_gateway_has_no_certificate(self) -> None: + run_spec = _service_run_spec(https="auto") + gw = _gateway_config(certificate=None) + assert _get_service_https(run_spec, gw) is False + + def test_auto_resolves_to_false_with_acm_gateway(self) -> None: + run_spec = _service_run_spec(https="auto") + gw = _gateway_config( + certificate=ACMGatewayCertificate(arn="arn:aws:acm:us-east-1:123:cert/abc") + ) + assert _get_service_https(run_spec, gw) is False + + def test_true_enables_https_regardless_of_gateway_certificate(self) -> None: + run_spec = _service_run_spec(https=True) + gw = _gateway_config(certificate=None) + assert _get_service_https(run_spec, gw) is True + + def test_false_disables_https_regardless_of_gateway_certificate(self) -> None: + run_spec = _service_run_spec(https=False) + gw = _gateway_config(certificate=LetsEncryptGatewayCertificate()) + assert _get_service_https(run_spec, gw) is False + + +class TestRegisterServiceInServerHttps: + def test_allows_default_true_without_gateway(self) -> None: + run_spec = _service_run_spec(https=True) + result = _register_service_in_server(_mock_run_model(), run_spec) + assert result is not None + + def test_allows_auto_without_gateway(self) -> None: + run_spec = _service_run_spec(https="auto") + result = _register_service_in_server(_mock_run_model(), run_spec) + assert result is not None + + def test_rejects_explicit_false_without_gateway(self) -> None: + run_spec = _service_run_spec(https=False) + with pytest.raises(ServerClientError, match="not applicable"): + _register_service_in_server(_mock_run_model(), run_spec)