diff --git a/.github/skills/migrate-backend-to-dts/SKILL.md b/.github/skills/migrate-backend-to-dts/SKILL.md new file mode 100644 index 0000000..665e5ee --- /dev/null +++ b/.github/skills/migrate-backend-to-dts/SKILL.md @@ -0,0 +1,607 @@ +--- +name: migrate-backend-to-dts +description: Migrate existing Azure Durable Functions apps from existing backend storage providers (Azure Storage, Netherite, MSSQL) to the Durable Task Scheduler. Use when switching backends, converting to azureManaged storage provider, upgrading from Azure Storage default provider, migrating from Netherite Event Hubs-based backend, migrating from Microsoft SQL Server backend, or modernizing Durable Functions infrastructure. Applies to .NET, Python, JavaScript/TypeScript, and Java Durable Functions apps that need to adopt the managed Durable Task Scheduler service. +--- + +# Migrate Durable Functions to Durable Task Scheduler + +Step-by-step guide for migrating Azure Durable Functions apps from existing backend storage providers to the Durable Task Scheduler (DTS). + +## Before You Start + +### ⚠️ Critical Prerequisites + +1. **Drain in-flight orchestrations.** DTS does not import state from other backends. All running orchestrations must complete or be terminated before switching. +2. **.NET apps must use isolated worker model.** DTS does not support the in-process hosting model. If your app uses in-process (`Microsoft.Azure.WebJobs.Extensions.DurableTask`), migrate to isolated worker first. +3. **Identity-based auth only.** DTS uses Microsoft Entra ID / managed identity — no shared keys or connection string secrets. Plan for RBAC setup. + +## Step 1: Identify Your Current Backend + +Inspect your `host.json` to determine which backend you're migrating from: + +| Current `storageProvider.type` | Backend | Key Indicator | +|-------------------------------|---------|---------------| +| *(missing or empty)* | **Azure Storage** (default) | No explicit type; uses `AzureWebJobsStorage` connection | +| `"azure"` | **Azure Storage** (explicit) | Same as default | +| `"netherite"` | **Netherite** | Requires Event Hubs connection string | +| `"mssql"` | **Microsoft SQL** | Requires SQL Server connection string | + +### Also check your packages for confirmation: + +**.NET (.csproj):** + +| Package | Backend | +|---------|---------| +| `Microsoft.Azure.WebJobs.Extensions.DurableTask` (no suffix) | Azure Storage (in-process — must also migrate to isolated) | +| `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` (no suffix) | Azure Storage (isolated) | +| `Microsoft.Azure.DurableTask.Netherite.AzureFunctions` | Netherite | +| `Microsoft.DurableTask.SqlServer.AzureFunctions` | MSSQL | + +**Python (requirements.txt):** `azure-functions-durable` — backend is configured in host.json only. + +**JavaScript (package.json):** `durable-functions` — backend is configured in host.json only. + +**Java (build.gradle/pom.xml):** `azure-functions-java-library` — backend is configured in host.json only. + +## Step 2: Update host.json + +Remove your old `storageProvider` block and replace it with the DTS configuration. + +### Migrating from Azure Storage (default) + +```json +// BEFORE — Azure Storage (default, no storageProvider block) +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub" + } + } +} + +// AFTER — Durable Task Scheduler +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +### Migrating from Azure Storage (explicit) + +```json +// BEFORE — Azure Storage (explicit type) +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub", + "storageProvider": { + "type": "azure", + "connectionStringName": "AzureWebJobsStorage" + } + } + } +} + +// AFTER — Durable Task Scheduler +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +### Migrating from Netherite + +```json +// BEFORE — Netherite +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub", + "storageProvider": { + "type": "netherite", + "storageConnectionName": "AzureWebJobsStorage", + "eventHubsConnectionName": "EventHubsConnection", + "partitionCount": 12 + } + } + } +} + +// AFTER — Durable Task Scheduler +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +> **Netherite cleanup:** After migration, remove the `EventHubsConnection` setting and consider deprovisioning the Event Hubs namespace if no longer needed. DTS handles partitioning internally — `partitionCount` is not needed. + +### Migrating from MSSQL + +```json +// BEFORE — Microsoft SQL Server +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub", + "storageProvider": { + "type": "mssql", + "connectionStringName": "SQLDB_Connection", + "taskEventLockTimeout": "00:02:00", + "createDatabaseIfNotExists": true, + "schemaName": "dt" + } + } + } +} + +// AFTER — Durable Task Scheduler +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +> **MSSQL cleanup:** After migration, remove `SQLDB_Connection` from app settings. The `dt.*` schema tables in your SQL database can be dropped once you've confirmed the migration is successful. + +### Non-.NET Languages (Python, JavaScript, Java) + +For Python, JavaScript, and Java apps, migration is **configuration-only** — no code changes or package changes are required. You only need to update `host.json`. + +There are two key differences from .NET: +1. The storage provider type is `"durabletask-scheduler"` (not `"azureManaged"`) +2. The extension bundle must be updated to the Preview bundle + +#### Python — Migrating from Azure Storage (default) + +```json +// BEFORE — host.json (Azure Storage default) +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub" + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.0.0, 5.0.0)" + } +} + +// AFTER — host.json (Durable Task Scheduler) +{ + "version": "2.0", + "logging": { + "logLevel": { + "DurableTask.Core": "Warning" + } + }, + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", + "version": "[4.29.0, 5.0.0)" + } +} +``` + +**requirements.txt** — no changes needed: +``` +azure-functions +azure-functions-durable +``` + +**local.settings.json:** +```json +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "FUNCTIONS_WORKER_RUNTIME": "python", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +#### JavaScript / TypeScript — Migrating from Azure Storage (default) + +```json +// BEFORE — host.json (Azure Storage default) +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub" + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.0.0, 5.0.0)" + } +} + +// AFTER — host.json (Durable Task Scheduler) +{ + "version": "2.0", + "logging": { + "logLevel": { + "DurableTask.Core": "Warning" + } + }, + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", + "version": "[4.29.0, 5.0.0)" + } +} +``` + +**package.json** — no changes needed: +```json +{ + "dependencies": { + "@azure/functions": "^4.0.0", + "durable-functions": "^3.0.0" + } +} +``` + +**local.settings.json:** +```json +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "FUNCTIONS_WORKER_RUNTIME": "node", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +#### Java — Migrating from Azure Storage (default) + +```json +// BEFORE — host.json (Azure Storage default) +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyTaskHub" + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.0.0, 5.0.0)" + } +} + +// AFTER — host.json (Durable Task Scheduler) +{ + "version": "2.0", + "logging": { + "logLevel": { + "DurableTask.Core": "Warning" + } + }, + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", + "version": "[4.29.0, 5.0.0)" + } +} +``` + +**pom.xml** — no changes needed. Your existing dependencies stay the same: +```xml + + com.microsoft.azure.functions + azure-functions-java-library + 3.2.3 + + + com.microsoft + durabletask-azure-functions + 1.7.0 + +``` + +**local.settings.json:** +```json +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "FUNCTIONS_WORKER_RUNTIME": "java", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None" + } +} +``` + +#### Non-.NET — Migrating from Netherite or MSSQL + +The target configuration is the same regardless of which existing backend you're migrating from. Replace the old `storageProvider` block and update the extension bundle as shown above. The only additional step is removing the old backend's connection strings from your app settings: + +- **From Netherite:** Remove `EventHubsConnection` (or your Event Hubs connection name) +- **From MSSQL:** Remove `SQLDB_Connection` (or your SQL connection name) + +## Step 3: Update Packages (.NET Only) + +Non-.NET languages do not need package changes — skip to Step 4. + +### Remove old backend package + +```xml + + + +``` + +### Add DTS backend package + +```xml + + + +``` + +### If migrating from in-process to isolated worker + +This is a larger migration. Replace the entire package set: + +```xml + + + + + + + + + + + + +``` + +> The in-process to isolated migration also requires code changes (new `Program.cs`, attribute changes, etc.). See [Microsoft's migration guide](https://learn.microsoft.com/azure/azure-functions/migrate-dotnet-to-isolated-model). + +## Step 4: Update Connection Strings + +### local.settings.json (local development) + +```json +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;Authentication=None", + "TASKHUB_NAME": "default" + } +} +``` + +### Azure App Settings (production) + +``` +DURABLE_TASK_SCHEDULER_CONNECTION_STRING=Endpoint=https://..durabletask.io;Authentication=DefaultAzure +TASKHUB_NAME= +``` + +### Remove old connection strings + +| Backend | Settings to Remove | +|---------|-------------------| +| Azure Storage | `AzureWebJobsStorage` is still needed for Functions runtime, but no longer used for Durable state | +| Netherite | `EventHubsConnection` (or your Event Hubs connection name) | +| MSSQL | `SQLDB_Connection` (or your SQL connection name) | + +## Step 5: Set Up Authentication + +DTS uses identity-based authentication. No shared keys. + +### Local Development + +No authentication needed — the emulator accepts unauthenticated requests: +``` +Endpoint=http://localhost:8080;Authentication=None +``` + +### Azure Production + +1. **Enable managed identity** on your Function App (system-assigned or user-assigned) +2. **Assign RBAC role** — grant the Function App's identity the `Durable Task Scheduler Task Hub Contributor` role on the DTS resource +3. **Use DefaultAzure authentication** in your connection string: + ``` + Endpoint=https://..durabletask.io;Authentication=DefaultAzure + ``` + +## Step 6: Handle Large Payloads (Optional) + +If your orchestrations pass large inputs/outputs (>10 KB), enable large payload storage to avoid message size limits: + +**.NET:** +```json +{ + "extensions": { + "durableTask": { + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING", + "largePayloadStorageEnabled": true, + "largePayloadStorageThresholdBytes": 10240 + }, + "hubName": "%TASKHUB_NAME%" + } + } +} +``` + +**Python / JavaScript / Java:** +```json +{ + "extensions": { + "durableTask": { + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING", + "largePayloadStorageEnabled": true, + "largePayloadStorageThresholdBytes": 10240 + }, + "hubName": "default" + } + } +} +``` + +This offloads large payloads to Azure Blob Storage via the `AzureWebJobsStorage` connection. + +## Step 7: Validate Locally + +```bash +# 1. Start the DTS emulator +docker run -d -p 8080:8080 -p 8082:8082 --name dts-emulator mcr.microsoft.com/dts/dts-emulator:latest + +# 2. Start Azurite (required for Azure Functions runtime) +azurite start + +# 3. Run your Function App +func start + +# 4. Open the DTS dashboard to monitor orchestrations +open http://localhost:8082 +``` + +Trigger your orchestrations and verify they complete successfully. Check the dashboard for execution history. + +## Step 8: Deploy to Azure + +1. **Provision a Durable Task Scheduler resource** in the Azure portal or via CLI +2. **Create a task hub** within the scheduler +3. **Configure managed identity** and RBAC (Step 5) +4. **Set app settings** with the production connection string (Step 4) +5. **Deploy your Function App** +6. **Verify** orchestrations run correctly in Azure using the [DTS dashboard](https://dashboard.durabletask.io) + +## Migration Warnings + +### ⚠️ No State Migration + +Running orchestrations do **not** carry over between backends. Before switching: +- Wait for all in-flight orchestrations to complete, **or** +- Terminate remaining orchestrations and accept they won't resume + +There is no tool to export/import orchestration state between backends. + +### ⚠️ In-Process Model Not Supported + +DTS only supports the **isolated worker model** for .NET. If your app uses `Microsoft.Azure.WebJobs.Extensions.DurableTask` (in-process), you must migrate to isolated worker first. See [Migrate to isolated worker](https://learn.microsoft.com/azure/azure-functions/migrate-dotnet-to-isolated-model). + +### ⚠️ Task Hub Names + +Task hub names from your old backend won't conflict with DTS — they are separate systems. You can use any name for your DTS task hub. + +### ⚠️ Netherite-Specific + +- Remove Event Hubs namespace connection strings +- Remove `partitionCount` configuration — DTS manages partitions automatically +- Event Hubs resources can be deprovisioned if only used for Netherite + +### ⚠️ MSSQL-Specific + +- Remove SQL connection strings from app settings +- The `dt.*` schema tables can be dropped after confirming successful migration +- `taskEventLockTimeout`, `createDatabaseIfNotExists`, and `schemaName` settings are not used by DTS + +### ⚠️ Custom Status and External Events + +These APIs (`SetCustomStatus`, `RaiseEventAsync`, `WaitForExternalEvent`) work identically on DTS. **No code changes needed.** + +### ⚠️ Durable Entities + +Durable Entities are supported on DTS with .NET isolated worker. No code changes needed — only the backend configuration changes. + +## Migration Checklist + +- [ ] All in-flight orchestrations drained or terminated +- [ ] .NET app uses isolated worker model (not in-process) +- [ ] `host.json` updated with `azureManaged` (or `durabletask-scheduler`) storage provider +- [ ] Extension bundle updated to Preview (non-.NET only) +- [ ] Old backend NuGet packages removed (.NET only) +- [ ] `Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureManaged` added (.NET only) +- [ ] `Azure.Identity` package added (.NET only) +- [ ] Old connection strings removed from app settings +- [ ] DTS connection string added to app settings +- [ ] DTS emulator tested locally +- [ ] Managed identity configured in Azure +- [ ] RBAC role assigned (`Durable Task Scheduler Task Hub Contributor`) +- [ ] Large payload storage configured (if needed) +- [ ] Deployed and validated in Azure + +## References + +- [Durable Task Scheduler overview](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/durable-task-scheduler) +- [Develop with Durable Task Scheduler (migration guide)](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/develop-with-durable-task-scheduler-functions) +- [Identity-based authentication](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/durable-task-scheduler-identity) +- [Migrate .NET to isolated worker model](https://learn.microsoft.com/azure/azure-functions/migrate-dotnet-to-isolated-model) +- [references/backends.md](references/backends.md) — Detailed backend comparison and configuration +- [references/setup.md](references/setup.md) — DTS provisioning, emulator, and identity setup diff --git a/.github/skills/migrate-backend-to-dts/references/backends.md b/.github/skills/migrate-backend-to-dts/references/backends.md new file mode 100644 index 0000000..63f204b --- /dev/null +++ b/.github/skills/migrate-backend-to-dts/references/backends.md @@ -0,0 +1,334 @@ +# Existing Backend Reference + +Detailed configuration reference for each existing Durable Functions backend, showing what to look for and what to remove during migration. + +## Azure Storage (Default Backend) + +The default backend when no `storageProvider` is configured. + +### How to Identify + +**host.json** — no `storageProvider` block, or `type` is `"azure"`: + +```json +{ + "extensions": { + "durableTask": { + "hubName": "MyTaskHub" + } + } +} +``` + +Or explicitly: + +```json +{ + "extensions": { + "durableTask": { + "hubName": "MyTaskHub", + "storageProvider": { + "type": "azure", + "connectionStringName": "AzureWebJobsStorage" + } + } + } +} +``` + +**Azure Storage-specific settings** you may see: + +```json +{ + "extensions": { + "durableTask": { + "storageProvider": { + "type": "azure", + "connectionStringName": "AzureWebJobsStorage", + "controlQueueBatchSize": 32, + "controlQueueBufferThreshold": 128, + "controlQueueVisibilityTimeout": "00:05:00", + "maxQueuePollingInterval": "00:00:30", + "partitionCount": 4, + "trackingStoreConnectionStringName": "TrackingStorage", + "trackingStoreNamePrefix": "DurableTask", + "useLegacyPartitionManagement": false, + "useTablePartitionManagement": true + } + } + } +} +``` + +**Packages (.NET):** +- In-process: `Microsoft.Azure.WebJobs.Extensions.DurableTask` +- Isolated: `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` (no AzureManaged suffix) + +**Azure Resources Used:** +- Azure Storage account (Tables + Queues + Blobs) + - Tables: `Instances`, `History` + - Queues: `-control-`, `-workitems` + - Blobs: `-leases` + +### What to Remove + +- Remove Storage-specific settings: `controlQueueBatchSize`, `controlQueueBufferThreshold`, `controlQueueVisibilityTimeout`, `maxQueuePollingInterval`, `partitionCount`, `trackingStoreConnectionStringName`, `trackingStoreNamePrefix`, `useLegacyPartitionManagement`, `useTablePartitionManagement` +- Remove `TrackingStorage` connection string (if separate tracking store was used) +- Keep `AzureWebJobsStorage` — still needed for the Azure Functions runtime + +### What to Replace With + +**.NET:** +```json +{ + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +**Python / JavaScript / Java:** +```json +{ + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", + "version": "[4.29.0, 5.0.0)" + } +} +``` + +--- + +## Netherite Backend + +High-throughput backend built on Azure Event Hubs and FASTER. + +### How to Identify + +**host.json:** + +```json +{ + "extensions": { + "durableTask": { + "hubName": "MyTaskHub", + "storageProvider": { + "type": "netherite", + "storageConnectionName": "AzureWebJobsStorage", + "eventHubsConnectionName": "EventHubsConnection", + "partitionCount": 12 + } + } + } +} +``` + +**Netherite-specific settings** you may see: + +```json +{ + "extensions": { + "durableTask": { + "storageProvider": { + "type": "netherite", + "storageConnectionName": "AzureWebJobsStorage", + "eventHubsConnectionName": "EventHubsConnection", + "partitionCount": 12, + "CacheDebugger": false, + "LogLevelLimit": "Information", + "StorageLogLevelLimit": "Warning", + "TransportLogLevelLimit": "Warning", + "EventLogLevelLimit": "Warning", + "WorkItemLogLevelLimit": "Warning", + "TakeStateCheckpointWhenStoppingPartition": true, + "MaxNumberBytesBetweenCheckpoints": 200000000, + "MaxTimeMsBetweenCheckpoints": 60000, + "IdleCheckpointFrequencyMs": 60000 + } + } + } +} +``` + +**Packages (.NET):** +- `Microsoft.Azure.DurableTask.Netherite.AzureFunctions` + +**Connection Strings:** +- `EventHubsConnection` — Azure Event Hubs namespace connection string +- `AzureWebJobsStorage` — Azure Storage account + +**Azure Resources Used:** +- Azure Event Hubs namespace (partition-per-partition mapping) +- Azure Storage account (for checkpoints and state via FASTER) + +### What to Remove + +- Remove `Microsoft.Azure.DurableTask.Netherite.AzureFunctions` NuGet package +- Remove `EventHubsConnection` from app settings +- Remove all Netherite-specific settings: `partitionCount`, `CacheDebugger`, `LogLevelLimit`, all `*LogLevelLimit` settings, `TakeStateCheckpointWhenStoppingPartition`, `MaxNumberBytesBetweenCheckpoints`, `MaxTimeMsBetweenCheckpoints`, `IdleCheckpointFrequencyMs` +- Consider deprovisioning Event Hubs namespace if only used by Netherite + +### What to Replace With + +**.NET:** +```json +{ + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +**Python / JavaScript / Java:** +```json +{ + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", + "version": "[4.29.0, 5.0.0)" + } +} +``` + +--- + +## Microsoft SQL Server Backend + +Backend using SQL Server for orchestration state storage. + +### How to Identify + +**host.json:** + +```json +{ + "extensions": { + "durableTask": { + "hubName": "MyTaskHub", + "storageProvider": { + "type": "mssql", + "connectionStringName": "SQLDB_Connection", + "createDatabaseIfNotExists": true + } + } + } +} +``` + +**MSSQL-specific settings** you may see: + +```json +{ + "extensions": { + "durableTask": { + "storageProvider": { + "type": "mssql", + "connectionStringName": "SQLDB_Connection", + "taskEventLockTimeout": "00:02:00", + "createDatabaseIfNotExists": true, + "databaseName": "DurableFunctionsDB", + "schemaName": "dt" + } + } + } +} +``` + +**Packages (.NET):** +- `Microsoft.DurableTask.SqlServer.AzureFunctions` + +**Connection Strings:** +- `SQLDB_Connection` — SQL Server connection string (e.g., `Server=...;Database=...;User ID=...;Password=...`) + +**Azure Resources Used:** +- Azure SQL Database or SQL Server + - Schema: `dt` (default) + - Tables: `dt.Instances`, `dt.History`, `dt.Payloads`, `dt.NewEvents`, `dt.NewTasks`, `dt.Versions`, `dt.GlobalSettings` + +### What to Remove + +- Remove `Microsoft.DurableTask.SqlServer.AzureFunctions` NuGet package +- Remove `SQLDB_Connection` from app settings +- Remove all MSSQL-specific settings: `taskEventLockTimeout`, `createDatabaseIfNotExists`, `databaseName`, `schemaName` +- After confirming successful migration, drop the `dt.*` schema tables from your SQL database + +### What to Replace With + +**.NET:** +```json +{ + "extensions": { + "durableTask": { + "hubName": "%TASKHUB_NAME%", + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + } +} +``` + +**Python / JavaScript / Java:** +```json +{ + "extensions": { + "durableTask": { + "hubName": "default", + "storageProvider": { + "type": "durabletask-scheduler", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING" + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview", + "version": "[4.29.0, 5.0.0)" + } +} +``` + +--- + +## Backend Comparison + +| Feature | Azure Storage | Netherite | MSSQL | Durable Task Scheduler | +|---------|--------------|-----------|-------|----------------------| +| **Transport** | Queue polling | Event Hubs streaming | SQL polling | gRPC push streaming | +| **State Store** | Table Storage | FASTER + Blob | SQL Server | Managed (built-in) | +| **Partitioning** | 4 (configurable) | Configurable (Event Hubs) | None | Managed (automatic) | +| **Auth Model** | Shared key / connection string | Shared key / connection string | SQL auth / Entra | Identity-only (Entra) | +| **Scaling** | Tied to partition count | Tied to Event Hubs partitions | Limited by SQL | Independent, automatic | +| **Dashboard** | None (use App Insights) | None | None | Built-in (dashboard.durabletask.io) | +| **Managed Service** | No (self-managed storage) | No (self-managed) | No (self-managed) | Yes (fully managed) | +| **Local Dev** | Azurite | Azurite + Event Hubs emulator | SQL Server | Docker emulator | +| **Large Payloads** | Blob overflow (automatic) | FASTER storage | SQL (limited) | Configurable blob offload | diff --git a/.github/skills/migrate-backend-to-dts/references/setup.md b/.github/skills/migrate-backend-to-dts/references/setup.md new file mode 100644 index 0000000..6e31f35 --- /dev/null +++ b/.github/skills/migrate-backend-to-dts/references/setup.md @@ -0,0 +1,292 @@ +# Durable Task Scheduler Setup Reference + +Complete guide for provisioning and configuring the Durable Task Scheduler for migrated Durable Functions apps. + +## Local Development with Emulator + +### Prerequisites + +```bash +# Docker (required for DTS emulator) +# Install from https://docs.docker.com/get-docker/ + +# Azure Functions Core Tools +brew tap azure/functions +brew install azure-functions-core-tools@4 + +# Azurite (Azure Storage emulator — still needed for Functions runtime) +npm install -g azurite +``` + +### Start the Emulator + +```bash +# Pull and run the DTS emulator +docker run -d -p 8080:8080 -p 8082:8082 --name dts-emulator mcr.microsoft.com/dts/dts-emulator:latest + +# Verify it's running +curl -s http://localhost:8082/health +``` + +- **Port 8080** — gRPC/HTTP endpoint (your app connects here) +- **Port 8082** — Dashboard UI (monitor orchestrations) +- **Dashboard:** http://localhost:8082 + +### Docker Compose (Emulator + Azurite) + +```yaml +# docker-compose.yml +version: '3.8' + +services: + azurite: + image: mcr.microsoft.com/azure-storage/azurite:latest + ports: + - "10000:10000" # Blob + - "10001:10001" # Queue + - "10002:10002" # Table + volumes: + - azurite-data:/data + command: azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 + + dts-emulator: + image: mcr.microsoft.com/dts/dts-emulator:latest + ports: + - "8080:8080" # gRPC/HTTP endpoint + - "8082:8082" # Dashboard + environment: + - DTS_EMULATOR_LOG_LEVEL=Information + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8082/health"] + interval: 10s + timeout: 5s + retries: 5 + +volumes: + azurite-data: +``` + +```bash +docker-compose up -d +``` + +### Local Connection Strings + +**local.settings.json:** + +```json +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING": "Endpoint=http://localhost:8080;Authentication=None", + "TASKHUB_NAME": "default" + } +} +``` + +## Azure Provisioning + +### Option 1: Azure Portal + +1. Go to the [Azure portal](https://portal.azure.com) +2. Search for "Durable Task Scheduler" in the marketplace +3. Create a new scheduler resource: + - Choose a **SKU** (Dedicated or Consumption) + - Select a **region** + - Provide a **name** +4. After creation, navigate to the resource and create a **Task Hub** +5. Copy the endpoint URL from the overview page + +### Option 2: Azure CLI + +```bash +# Install the Durable Task Scheduler extension (if not already installed) +az extension add --name durabletask + +# Create a scheduler +az durabletask scheduler create \ + --resource-group \ + --name \ + --location \ + --sku dedicated + +# Create a task hub +az durabletask taskhub create \ + --resource-group \ + --scheduler-name \ + --name + +# Get the endpoint +az durabletask scheduler show \ + --resource-group \ + --name \ + --query endpoint -o tsv +``` + +### Option 3: Bicep + +```bicep +resource scheduler 'Microsoft.DurableTask/schedulers@2025-04-01-preview' = { + name: schedulerName + location: location + properties: { + sku: { + name: 'Dedicated' + capacity: 1 + } + } +} + +resource taskHub 'Microsoft.DurableTask/schedulers/taskHubs@2025-04-01-preview' = { + parent: scheduler + name: taskHubName +} +``` + +## Identity & Authentication + +DTS uses identity-based authentication exclusively. No shared keys or secret-bearing connection strings. + +### Configure Managed Identity + +```bash +# Enable system-assigned managed identity on your Function App +az functionapp identity assign \ + --resource-group \ + --name + +# Get the principal ID +PRINCIPAL_ID=$(az functionapp identity show \ + --resource-group \ + --name \ + --query principalId -o tsv) +``` + +### Assign RBAC Role + +```bash +# Get the DTS scheduler resource ID +SCHEDULER_ID=$(az durabletask scheduler show \ + --resource-group \ + --name \ + --query id -o tsv) + +# Assign the Task Hub Contributor role +az role assignment create \ + --assignee "$PRINCIPAL_ID" \ + --role "Durable Task Scheduler Task Hub Contributor" \ + --scope "$SCHEDULER_ID" +``` + +### Connection String Format + +| Environment | Connection String | +|-------------|-------------------| +| **Local emulator** | `Endpoint=http://localhost:8080;Authentication=None` | +| **Azure (managed identity)** | `Endpoint=https://..durabletask.io;Authentication=DefaultAzure` | +| **Azure (specific task hub)** | `Endpoint=https://..durabletask.io;TaskHub=;Authentication=DefaultAzure` | + +### Set App Settings + +```bash +az functionapp config appsettings set \ + --resource-group \ + --name \ + --settings \ + "DURABLE_TASK_SCHEDULER_CONNECTION_STRING=Endpoint=https://..durabletask.io;Authentication=DefaultAzure" \ + "TASKHUB_NAME=" +``` + +## Large Payload Storage + +DTS has a message size limit. For orchestrations that pass large inputs/outputs, enable blob-based overflow: + +### Configuration + +```json +{ + "extensions": { + "durableTask": { + "storageProvider": { + "type": "azureManaged", + "connectionStringName": "DURABLE_TASK_SCHEDULER_CONNECTION_STRING", + "largePayloadStorageEnabled": true, + "largePayloadStorageThresholdBytes": 10240 + }, + "hubName": "%TASKHUB_NAME%" + } + } +} +``` + +- **`largePayloadStorageEnabled`** — set to `true` to enable blob offload +- **`largePayloadStorageThresholdBytes`** — payloads larger than this (default 10240 = 10 KB) are stored in blob + +Large payloads are stored in the Azure Storage account referenced by `AzureWebJobsStorage`. + +## Monitoring + +### DTS Dashboard + +- **Azure:** [dashboard.durabletask.io](https://dashboard.durabletask.io) — view orchestration status, history, and perform management operations +- **Local emulator:** http://localhost:8082 + +### Application Insights + +Durable Functions continues to emit telemetry to Application Insights when configured. No changes needed for monitoring integration. + +### Distributed Tracing + +DTS supports distributed tracing v2 with Application Insights. Enable in host.json: + +```json +{ + "extensions": { + "durableTask": { + "tracing": { + "distributedTracingEnabled": true, + "distributedTracingProtocol": "W3CTraceContext" + } + } + } +} +``` + +## Troubleshooting + +### Emulator Won't Start + +```bash +# Check if ports are in use +lsof -i :8080 +lsof -i :8082 + +# Pull latest image +docker pull mcr.microsoft.com/dts/dts-emulator:latest + +# Remap ports if needed +docker run -d -p 9080:8080 -p 9082:8082 --name dts-emulator mcr.microsoft.com/dts/dts-emulator:latest +# Update connection string: Endpoint=http://localhost:9080;Authentication=None +``` + +### Authentication Errors in Azure + +```bash +# Verify managed identity is enabled +az functionapp identity show --resource-group --name + +# Verify role assignment exists +az role assignment list --assignee --scope + +# Check the role name is correct +# Must be: "Durable Task Scheduler Task Hub Contributor" +``` + +### Orchestrations Not Starting + +1. Verify `host.json` has correct `storageProvider.type` (`azureManaged` for .NET, `durabletask-scheduler` for other languages) +2. Verify connection string is accessible (check app settings) +3. Check Function App logs for connection errors +4. Verify task hub exists in the scheduler resource +5. For non-.NET: verify extension bundle is `Microsoft.Azure.Functions.ExtensionBundle.Preview` with version `[4.29.0, 5.0.0)`