diff --git a/.github/instructions/ado-pipelines.instructions.md b/.github/instructions/ado-pipelines.instructions.md index 82b8046dee..db053130e0 100644 --- a/.github/instructions/ado-pipelines.instructions.md +++ b/.github/instructions/ado-pipelines.instructions.md @@ -1,195 +1,121 @@ --- applyTo: "eng/pipelines/**/*.yml" --- -# Azure DevOps Pipelines Guide - -## Overview - -This repository uses Azure DevOps Pipelines for CI/CD. The pipeline configurations are located in `eng/pipelines/`. - -**ADO Organization**: sqlclientdrivers -**ADO Project**: ADO.NET - -## Pipeline Structure - -``` -eng/pipelines/ -├── abstractions/ # Abstractions package pipelines -├── azure/ # Azure package pipelines -├── common/ # Shared templates -│ └── templates/ -│ ├── jobs/ # Reusable job templates -│ ├── stages/ # Reusable stage templates -│ └── steps/ # Reusable step templates -├── jobs/ # Top-level job definitions -├── libraries/ # Shared variable definitions -├── stages/ # Stage definitions -├── steps/ # Step definitions -├── variables/ # Variable templates -├── akv-official-pipeline.yml # AKV provider official/signing build -├── dotnet-sqlclient-ci-core.yml # Core CI pipeline (reusable) -├── dotnet-sqlclient-ci-package-reference-pipeline.yml # CI with package references -├── dotnet-sqlclient-ci-project-reference-pipeline.yml # CI with project references -├── dotnet-sqlclient-signing-pipeline.yml # Package signing pipeline -├── sqlclient-pr-package-ref-pipeline.yml # PR validation (package ref) -├── sqlclient-pr-project-ref-pipeline.yml # PR validation (project ref) -└── stress-tests-pipeline.yml # Stress testing -``` - -## Main Pipelines - -### CI Core Pipeline (`dotnet-sqlclient-ci-core.yml`) -Reusable core CI pipeline consumed by both project-reference and package-reference CI pipelines. Configurable parameters: - -| Parameter | Description | Default | -|-----------|-------------|---------| -| `targetFrameworks` | Windows test frameworks | `[net462, net8.0, net9.0, net10.0]` | -| `targetFrameworksUnix` | Unix test frameworks | `[net8.0, net9.0, net10.0]` | -| `referenceType` | Project or Package reference | Required | -| `buildConfiguration` | Debug or Release | Required | -| `useManagedSNI` | Test with managed SNI | `[false, true]` | -| `testJobTimeout` | Test job timeout (minutes) | Required | -| `runAlwaysEncryptedTests` | Include AE tests | `true` | -| `enableStressTests` | Include stress test stage | `false` | - -### CI Reference Pipelines -- `dotnet-sqlclient-ci-project-reference-pipeline.yml` — Full CI using project references (builds from source) -- `dotnet-sqlclient-ci-package-reference-pipeline.yml` — Full CI using package references (tests against published NuGet packages) - -### PR Validation Pipelines -- `sqlclient-pr-project-ref-pipeline.yml` — PR validation with project references -- `sqlclient-pr-package-ref-pipeline.yml` — PR validation with package references - -These pipelines trigger on pull requests and run a subset of the full CI matrix to provide fast feedback. - -### Official/Signing Pipeline (`dotnet-sqlclient-signing-pipeline.yml`) -Signs and publishes NuGet packages. Used for official releases. Requires secure service connections and key vault access for code signing. - -### AKV Official Pipeline (`akv-official-pipeline.yml`) -Builds and signs the `Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider` add-on package separately from the main driver. Uses 1ES pipeline templates for compliance. - -### Stress Tests Pipeline (`stress-tests-pipeline.yml`) -Optional pipeline for long-running stress and endurance testing. Enabled via `enableStressTests` parameter in CI core. - -## Build Stages - -1. **build_abstractions_package_stage**: Build and pack abstractions -2. **build_sqlclient_package_stage**: Build main driver and AKV packages -3. **build_azure_package_stage**: Build Azure extensions package -4. **stress_tests_stage**: Optional stress testing -5. **run_tests_stage**: Execute all test suites +# Azure DevOps CI/CD Pipeline Guidelines + +## Purpose + +Rules and conventions for editing the Azure DevOps CI/CD pipelines that build, test, and validate Microsoft.Data.SqlClient. These pipelines live under `eng/pipelines/` (excluding `onebranch/`, which is covered by separate instructions). + +**ADO Organization**: sqlclientdrivers | **ADO Project**: ADO.NET + +## Pipeline Layout + +Two categories of pipelines exist in this repository: + +- **CI/PR pipelines** (`eng/pipelines/`) — Build, test, and validate on every push/PR +- **OneBranch pipelines** (`eng/pipelines/onebranch/`) — Official signing/release builds (separate instructions file) + +Top-level CI/PR pipeline files: +- `dotnet-sqlclient-ci-core.yml` — Reusable core template; all CI and PR pipelines extend this +- `dotnet-sqlclient-ci-package-reference-pipeline.yml` — CI with Package references (Release) +- `dotnet-sqlclient-ci-project-reference-pipeline.yml` — CI with Project references (Release) +- `sqlclient-pr-package-ref-pipeline.yml` — PR validation with Package references +- `sqlclient-pr-project-ref-pipeline.yml` — PR validation with Project references +- `stress-tests-pipeline.yml` — Stress tests triggered after successful CI-Package runs + +Reusable templates are organized under: +- `common/templates/jobs/` — Job templates (`ci-build-nugets-job`, `ci-code-coverage-job`, `ci-run-tests-job`) +- `common/templates/stages/` — Stage templates (`ci-run-tests-stage`) +- `common/templates/steps/` — Step templates (build, test, config, publish) +- `jobs/` — Package-specific CI jobs (pack/test Abstractions, Azure, Logging, stress) +- `stages/` — Package-specific CI stages (build Logging → Abstractions → SqlClient → Azure → verify → stress) +- `libraries/` — Shared variables (`ci-build-variables.yml`) +- `steps/` — SDK install steps + +## CI Core Template + +`dotnet-sqlclient-ci-core.yml` is the central orchestrator. All CI and PR pipelines extend it with different parameters. + +Key parameters: +- `referenceType` (required) — `Package` or `Project`; controls how sibling packages are referenced +- `buildConfiguration` (required) — `Debug` or `Release` +- `testJobTimeout` (required) — test job timeout in minutes +- `targetFrameworks` — Windows test TFMs; default `[net462, net8.0, net9.0, net10.0]` +- `targetFrameworksUnix` — Unix test TFMs; default `[net8.0, net9.0, net10.0]` +- `testSets` — test partitions; default `[1, 2, 3]` +- `useManagedSNI` — SNI variants to test; default `[false, true]` +- `runAlwaysEncryptedTests` — include AE test set; default `true` +- `enableStressTests` — enable stress test stage; default `false` +- `debug` — enable debug output; default `false` +- `dotnetVerbosity` — MSBuild verbosity; default `normal` + +## Build Stage Order + +Stages execute in dependency order (Package reference mode requires artifacts from prior stages): +1. `generate_secrets` — Generate test secrets +2. `build_logging_package_stage` — Build Logging package +3. `build_abstractions_package_stage` — Build Abstractions (depends on Logging) +4. `build_sqlclient_package_stage` — Build SqlClient + AKV Provider (depends on Abstractions + Logging) +5. `build_azure_package_stage` — Build Azure extensions (depends on Abstractions + Logging + SqlClient) +6. `verify_nuget_packages_stage` — Verify NuGet package metadata +7. `stress_tests_stage` — Optional stress tests +8. `ci_run_tests_stage` — Run MDS and AKV test suites + +When adding a new build stage, respect the dependency graph and pass artifact names/versions to downstream stages. + +## PR vs CI Pipeline Differences + +PR pipelines: +- Trigger on PRs to `dev/*`, `feat/*`, `main`; exclude `eng/pipelines/onebranch/*` paths +- Use reduced TFM matrix: `[net462, net8.0, net9.0]` (excludes net10.0) +- Timeout: 90 minutes +- Package-ref PR disables Always Encrypted tests in Debug config + +CI pipelines: +- Trigger on push to `main` (GitHub) and `internal/main` (ADO) with `batch: true` +- Scheduled weekday builds (see individual pipeline files for cron times) +- Full TFM matrix including net10.0 ## Test Configuration -### Test Sets -Tests are divided into sets for parallelization: -- `TestSet=1` — First partition of tests -- `TestSet=2` — Second partition -- `TestSet=3` — Third partition -- `TestSet=AE` — Always Encrypted tests - -### Test Filters -Tests use category-based filtering. The default filter excludes both `failing` and `flaky` tests: -``` -category!=failing&category!=flaky -``` - -Category values: -- `nonnetfxtests` — Excluded on .NET Framework -- `nonnetcoreapptests` — Excluded on .NET Core -- `nonwindowstests` — Excluded on Windows -- `nonlinuxtests` — Excluded on Linux -- `failing` — Known permanent failures (excluded from all runs) -- `flaky` — Intermittently failing tests (quarantined, run separately) - -### Flaky Test Quarantine in Pipelines -Quarantined tests (`[Trait("Category", "flaky")]`) run in **separate pipeline steps** after the main test steps. This ensures: -- Main test runs are **not blocked** by intermittent failures -- Flaky tests are still **monitored** for regression or resolution -- Code coverage is **not collected** for flaky test runs -- Results appear in pipeline output for visibility - -The quarantine steps are configured in: -- `eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml` -- `eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml` -- `eng/pipelines/common/templates/steps/run-all-tests-step.yml` - -### Test Timeout -All test runs use `--blame-hang-timeout 10m` (configured in `build.proj`). Tests exceeding 10 minutes are killed and reported as failures. - -### SNI Testing -The `useManagedSNI` parameter controls testing with: -- Native SNI (`false`) - Windows native library -- Managed SNI (`true`) - Cross-platform managed implementation +Test partitioning: +- Tests split into `TestSet=1`, `TestSet=2`, `TestSet=3` for parallelization +- `TestSet=AE` — Always Encrypted tests (controlled by `runAlwaysEncryptedTests`) -## Variables - -### Build Variables (`ci-build-variables.yml`) -Common build configuration: -- Package versions -- Build paths -- Signing configuration - -### Runtime Variables -Set via pipeline parameters or UI: -- `Configuration` - Debug/Release -- `Platform` - AnyCPU/x86/x64 -- `TF` - Target framework - -## Creating Pipeline Changes - -### Adding New Test Categories -1. Add category attribute to tests: `[Category("newcategory")]` -2. Update filter expressions in test job templates -3. Document category purpose in test documentation - -### Adding New Pipeline Parameters -1. Define parameter in appropriate `.yml` file -2. Add to parameter passing in calling templates -3. Document in this file - -### Modifying Build Steps -1. Changes should be made in template files for reusability -2. Test changes locally when possible -3. Submit as PR - validation will run +Test filters — default excludes `failing` and `flaky` categories: +- `failing` — known permanent failures, always excluded +- `flaky` — intermittent failures, quarantined in separate pipeline steps +- `nonnetfxtests` / `nonnetcoreapptests` — platform-specific exclusions +- `nonwindowstests` / `nonlinuxtests` — OS-specific exclusions -## Best Practices +Flaky test quarantine: +- Quarantined tests (`[Trait("Category", "flaky")]`) run in separate steps after main tests +- Main test runs are not blocked by flaky failures +- No code coverage collected for flaky runs +- Configured in `common/templates/steps/build-and-run-tests-netcore-step.yml`, `build-and-run-tests-netfx-step.yml`, and `run-all-tests-step.yml` -### Template Design -- Use templates for reusable definitions -- Pass parameters explicitly (avoid global variables) -- Use descriptive stage/job/step names +SNI testing — `useManagedSNI` controls testing with native SNI (`false`) or managed SNI (`true`) -### Variable Management -- Use template variables for shared values -- Use pipeline parameters for per-run configuration -- Avoid hardcoding versions (use Directory.Packages.props) +Test timeout — `--blame-hang-timeout 10m` (configured in `build.proj`); tests exceeding 10 minutes are killed -### Test Infrastructure -- Ensure tests are properly categorized -- Handle test configuration files properly -- Use test matrix for cross-platform coverage - -## Troubleshooting - -### Common Issues -1. **Test failures due to missing config**: Ensure `config.json` exists -2. **Platform-specific failures**: Check platform exclusion categories -3. **Timeout issues**: Increase `testJobTimeout` parameter - -### Debugging Pipelines -- Enable debug mode via `debug: true` parameter -- Use `dotnetVerbosity: diagnostic` for detailed output -- Check build logs in Azure DevOps - -## Security Considerations - -- Pipelines use service connections for artifact publishing -- Signing uses secure key vault integration -- Sensitive configuration should use pipeline secrets -- Never commit credentials in pipeline files - -## Related Documentation +## Variables -- [BUILDGUIDE.md](../../BUILDGUIDE.md) - Local build instructions -- [Azure DevOps Documentation](https://learn.microsoft.com/azure/devops/pipelines/) +- All CI build variables centralized in `libraries/ci-build-variables.yml` +- Package versions use `-ci` suffix (e.g., `7.0.0.$(Build.BuildNumber)-ci`) +- `assemblyBuildNumber` derived from first segment of `Build.BuildNumber` (16-bit safe) +- `localFeedPath` = `$(Build.SourcesDirectory)/packages` — local NuGet feed for inter-package deps +- `packagePath` = `$(Build.SourcesDirectory)/output` — NuGet pack output + +## Conventions When Editing Pipelines + +- Always use templates for reusable logic — do not inline complex steps +- Pass parameters explicitly; avoid relying on global variables +- Use descriptive stage/job/step display names +- When adding parameters, define them in the core template and thread through calling pipelines +- When adding test categories, update filter expressions in test step templates +- PR pipelines should run a minimal matrix for fast feedback +- Test changes via PR pipeline first — validation runs automatically +- Enable `debug: true` and `dotnetVerbosity: diagnostic` for troubleshooting +- Never commit credentials or secrets in pipeline files +- Signing and release are handled by OneBranch pipelines — not these CI/PR pipelines diff --git a/.github/instructions/onebranch-pipeline-design.instructions.md b/.github/instructions/onebranch-pipeline-design.instructions.md index 1c848e2493..f2319ee5b2 100644 --- a/.github/instructions/onebranch-pipeline-design.instructions.md +++ b/.github/instructions/onebranch-pipeline-design.instructions.md @@ -1,534 +1,139 @@ --- applyTo: "eng/pipelines/**/*.yml" --- -# Multi-Product Azure DevOps Pipeline in dotnet/sqlclient — Design Specification +# OneBranch Pipeline Guidelines -## 1. Overview +## Purpose -This document describes the design of the unified Azure DevOps YAML pipeline that builds, signs, packages, and optionally releases six NuGet packages with interdependencies. The pipeline uses **stages** and **jobs** to maximize parallelism while respecting dependency order. It comprises five stages: three build stages, a validation stage, and an on-demand release stage. +Rules and conventions for editing the OneBranch Azure DevOps YAML pipelines that build, sign, package, and release six NuGet packages with interdependencies. -Two pipeline variants exist from the same stage/job structure: +## Pipeline Variants + +- `sqlclient-official.yml` — Official pipeline; uses `OneBranch.Official.CrossPlat.yml`; CI trigger on `internal/main` + daily schedule at 04:30 UTC +- `sqlclient-non-official.yml` — Non-Official pipeline; uses `OneBranch.NonOfficial.CrossPlat.yml`; manual only (`pr: none`, `trigger: none`) +- Both live under `eng/pipelines/onebranch/` and extend OneBranch governed templates +- Never parameterize the OneBranch template name — hardcode it per pipeline for PRC compliance +- Official pipeline must never be run on PRs or dev branches. -| Pipeline | Template | Trigger | Purpose | -|----------|----------|---------|---------| -| `dotnet-sqlclient-official-pipeline.yml` | `OneBranch.Official.CrossPlat.yml` | CI + scheduled | Production-signed builds | -| `dotnet-sqlclient-non-official-pipeline.yml` | `OneBranch.NonOfficial.CrossPlat.yml` | Manual only | Validation / test builds (release in dry-run mode) | - -Both pipelines use the **OneBranch (1ES) governed template** infrastructure and share identical stage definitions, job templates, and variable chains. - ---- - -## 2. Products and Dependencies - -| # | Package | Dependencies | -|---|---------|-------------| -| 1 | `Microsoft.SqlServer.Server` | — | -| 2 | `Microsoft.Data.SqlClient.Internal.Logging` | — | -| 3 | `Microsoft.Data.SqlClient.Extensions.Abstractions` | `Internal.Logging` | -| 4 | `Microsoft.Data.SqlClient` | `Internal.Logging`, `Extensions.Abstractions` | -| 5 | `Microsoft.Data.SqlClient.Extensions.Azure` | `Extensions.Abstractions`, `Internal.Logging` | -| 6 | `Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider` | `SqlClient`, `Internal.Logging` | - ---- - -## 3. Pipeline Flow — Sequence Diagram - -```mermaid -sequenceDiagram - participant T as Trigger / User - participant P as Pipeline Orchestrator - participant B1a as Job: Build Internal.Logging - participant B1c as Job: Build SqlServer.Server - participant B1b as Job: Build Extensions.Abstractions - participant B2a as Job: Build SqlClient - participant B2b as Job: Build Extensions.Azure - participant V as Job: Validate MDS Package - participant B3 as Job: Build AKV Provider - participant R as Stage: Release - - Note over T,R: ══════ BUILD & SIGN PHASE ══════ - - T->>P: Pipeline triggered (CI / Scheduled / Manual) - - Note over P: Stage 1 — build_independent (parallel, no deps) - - par Stage 1 jobs (parallel) - P->>B1a: Build DLLs → ESRP sign DLLs → Pack → ESRP sign NuGet (Logging) - B1a-->>P: ✅ Signed .nupkg - and - P->>B1c: Build + ESRP sign + pack SqlServer.Server - B1c-->>P: ✅ Signed .nupkg - end - - Note over P: Stage 2 — build_abstractions (dependsOn: build_independent) - - P->>B1b: Build DLLs → ESRP sign DLLs → Pack → ESRP sign NuGet (Abstractions) - Note right of B1b: Downloads: Internal.Logging artifact - B1b-->>P: ✅ Signed .nupkg - - Note over P: Stage 3 — build_dependent (dependsOn: build_abstractions) - - par Stage 3 jobs (parallel) - P->>B2a: Build + ESRP sign + pack SqlClient - Note right of B2a: Downloads: Internal.Logging,
Extensions.Abstractions artifacts - B2a-->>P: ✅ Signed .nupkg + .snupkg - and - P->>B2b: Build DLLs → ESRP sign DLLs → Pack → ESRP sign NuGet (Azure) - Note right of B2b: Downloads: Extensions.Abstractions,
Internal.Logging artifacts - B2b-->>P: ✅ Signed .nupkg - end - - Note over P: Validation + Stage 4 (both dependsOn: build_dependent, run in parallel) - - par Validation and Stage 4 (parallel) - P->>V: Validate signed MDS package - V-->>P: ✅ Package validation passed - and - P->>B3: Build + ESRP sign + pack AKV Provider - Note right of B3: Downloads: SqlClient,
Internal.Logging artifacts - B3-->>P: ✅ Signed .nupkg - end - - Note over T,R: ══════ RELEASE PHASE (on-demand) ══════ - - alt At least one release parameter is true - P->>R: Stage: release (dependsOn: conditional on build stages) - Note right of R: ADO Environment Approval
(NuGet-Production environment) - R-->>P: ✅ Approved - Note right of R: Publish selected packages
via NuGetCommand@2 - R-->>P: ✅ Published to NuGet - else No release parameters set - Note over P: Release stage skipped - end - - Note over T,R: Pipeline complete 🎉 -``` - ---- - -## 4. Stage Design - -### 4.1 Build Phase - -The build phase runs automatically on every CI trigger, scheduled run, or manual queue. It is divided into four build stages plus a validation stage, based on the dependency graph. - -#### Stage 1 — `build_independent`: Independent Packages (no dependencies) - -| Job Template | Package | Build Target | Condition | -|--------------|---------|--------------|-----------| -| `build-signed-csproj-package-job.yml` | `Microsoft.Data.SqlClient.Internal.Logging` | `BuildLogging` / `PackLogging` | `buildAKVProvider OR buildSqlClient` | -| `build-signed-csproj-package-job.yml` | `Microsoft.SqlServer.Server` | `PackSqlServer` | `buildSqlServerServer` | - -- **`dependsOn`**: none -- **Parallelism**: Jobs run in parallel (depending on which are enabled) -- **Conditional builds**: Each job is wrapped with compile-time `${{ if }}` conditionals based on build parameters -- csproj-based jobs (`build-signed-csproj-package-job.yml`) perform: **Build DLLs → ESRP DLL signing → NuGet pack (NoBuild=true) → ESRP NuGet signing** → publish artifact - -#### Stage 2 — `build_abstractions`: Abstractions Package (depends on Stage 1) - -| Job Template | Package | Build Target | Artifact Dependencies | -|--------------|---------|--------------|----------------------| -| `build-signed-csproj-package-job.yml` | `Microsoft.Data.SqlClient.Extensions.Abstractions` | `BuildAbstractions` / `PackAbstractions` | `Internal.Logging` | - -- **Stage condition**: `buildSqlClient = true` (entire stage is excluded when false) -- **`dependsOn`**: `build_independent` -- Downloads `Microsoft.Data.SqlClient.Internal.Logging.nupkg` (from Stage 1) pipeline artifact - -#### Stage 3 — `build_dependent`: Core Packages (depend on Stage 2) - -| Job Template | Package | Build Target | Artifact Dependencies | -|--------------|---------|--------------|----------------------| -| `build-signed-package-job.yml` | `Microsoft.Data.SqlClient` | *(nuspec-based)* | `Internal.Logging`, `Extensions.Abstractions` | -| `build-signed-csproj-package-job.yml` | `Microsoft.Data.SqlClient.Extensions.Azure` | `BuildAzure` / `PackAzure` | `Extensions.Abstractions`, `Internal.Logging` | - -- **Stage condition**: `buildSqlClient = true` (entire stage is excluded when false) -- **`dependsOn`**: `build_abstractions` -- **Parallelism**: Both jobs run in parallel -- The MDS (SqlClient) job also publishes symbol packages (`.snupkg`) when `publishSymbols` is true -- All jobs configure APIScan with job-level `ob_sdl_apiscan_*` variables targeting package-specific folders - -#### Stage 4 — `build_addons`: Add-on Packages (depend on Stage 3) - -| Job Template | Package | Artifact Dependencies | -|--------------|---------|----------------------| -| `build-akv-official-job.yml` | `Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider` | `SqlClient`, `Internal.Logging` | - -- **Stage condition**: `buildAKVProvider AND buildSqlClient` (both must be true) -- **`dependsOn`**: `build_dependent` -- Downloads `Microsoft.Data.SqlClient.nupkg` (from Stage 3) and `Microsoft.Data.SqlClient.Internal.Logging.nupkg` (from Stage 1) pipeline artifacts -- Uses separate ESRP signing credentials (`Signing`-prefixed variables from `esrp-variables-v2` group) - -### 4.2 Validation Stage — `mds_package_validation` - -Validates the signed MDS (SqlClient) package after Stage 3 completes. - -- **Stage condition**: `buildSqlClient = true` -- **`dependsOn`**: `build_dependent` -- Runs in parallel with Stage 4 (`build_addons`) -- Uses `validate-signed-package-job.yml` template -- Downloads the `drop_build_dependent_build_signed_package` artifact and validates against `CurrentNetFxVersion` (default: `net462`) - -### 4.3 Release Phase — `release` - -The release stage is gated and only executes on demand when at least one release parameter is set to `true` at queue time. - -- **`dependsOn`**: Conditional based on which build stages are enabled: - - `build_independent` (when releasing SqlServer.Server or Logging) - - `build_abstractions` (when releasing Abstractions) - - `build_dependent`, `mds_package_validation` (when `buildSqlClient = true`) - - `build_addons` (when `buildAKVProvider AND buildSqlClient`) -- **Gate**: ADO Environment approvals (official pipeline only): - - Official: `NuGet-Production` environment with configured approvals - - Non-Official: `NuGet-DryRun` environment (no approvals, validation only) -- **Package selection**: Controlled by 6 runtime boolean parameters (see Section 5.2) -- **Stage condition**: The entire stage is skipped unless at least one release parameter is `true`: - ```yaml - - ${{ if or(parameters.releaseSqlServerServer, parameters.releaseLogging, ...) }}: - - stage: release - ``` -- **Publish jobs**: Each package has a conditional publish job that is included at compile time only when its parameter is `true`: - ```yaml - - ${{ if eq(parameters.releaseXxx, true) }}: - - template: /eng/pipelines/onebranch/jobs/publish-nuget-package-job.yml@self - ``` -- **Environment variables**: Stage sets `ob_release_usedeploymentjob: true` for OneBranch integration: - - Official: `ob_release_environment: 'NuGet-Production'` - - Non-Official: `ob_release_environment: 'NuGet-DryRun'` - -#### Artifact → Publish Job Mapping - -| Package | Artifact Name | Publish Job | -|---------|---------------|-------------| -| `Microsoft.SqlServer.Server` | `drop_build_independent_build_package_SqlServer` | `publish_SqlServer_Server` | -| `Microsoft.Data.SqlClient.Internal.Logging` | `drop_build_independent_build_package_Logging` | `publish_Logging` | -| `Microsoft.Data.SqlClient.Extensions.Abstractions` | `drop_build_abstractions_build_package_Abstractions` | `publish_Abstractions` | -| `Microsoft.Data.SqlClient` | `drop_build_dependent_build_package_SqlClient` | `publish_SqlClient` | -| `Microsoft.Data.SqlClient.Extensions.Azure` | `drop_build_dependent_build_package_Azure` | `publish_Extensions_Azure` | -| `Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider` | `drop_build_addons_buildSignedAkvPackage` | `publish_AKVProvider` | - -Each publish job uses the reusable `publish-nuget-package-job.yml` template, which downloads the artifact and pushes `.nupkg`/`.snupkg` files via `NuGetCommand@2` with an external feed service connection. - -#### Dry-Run Mode - -Two ADO environments control release behavior: - -| Environment | Pipeline | Behavior | -|------------|----------|----------| -| `NuGet-DryRun` | Non-Official | Validation only — packages are never pushed | -| `NuGet-Production` | Official | Real releases with approval gate | - -**Non-official pipeline**: Always runs in dry-run mode. There is no `releaseDryRun` parameter — `dryRun: true` is hardcoded in every publish job. This prevents accidental publication from validation builds. - -**Official pipeline**: Exposes a `releaseDryRun` parameter (default: `true` for safety). When enabled, the template downloads artifacts and lists the `.nupkg`/`.snupkg` files that *would* be published but skips the actual `NuGetCommand@2` push. Set `releaseDryRun: false` to perform real pushes after final validation. - ---- - -## 5. Runtime Parameters - -### 5.1 Build Parameters - -The pipeline exposes the following parameters at queue time: - -```yaml -parameters: - - name: debug - displayName: 'Enable debug output' - type: boolean - default: false - - - name: publishSymbols - displayName: 'Publish symbols' - type: boolean - default: false - - - name: CurrentNetFxVersion - displayName: 'Lowest supported .NET Framework version (MDS validation)' - type: string - default: 'net462' - - - name: isPreview - displayName: 'Is this a preview build?' - type: boolean - default: false - - - name: testJobTimeout - displayName: 'Test job timeout (in minutes)' - type: number - default: 60 - - # Build parameters — control which packages to build - - name: buildSqlServerServer - displayName: 'Build Microsoft.SqlServer.Server' - type: boolean - default: true - - - name: buildSqlClient - displayName: 'Build Microsoft.Data.SqlClient and Extensions' - type: boolean - default: true - - - name: buildAKVProvider - displayName: 'Build Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider' - type: boolean - default: true -``` - -The `isPreview` parameter controls version resolution — when `true`, each package uses its preview version (e.g., `loggingPackagePreviewVersion`) instead of the GA version (e.g., `loggingPackageVersion`). All versions are defined in the centralized `libraries/common-variables.yml`. - -The build parameters enable selective package building: -- `buildSqlServerServer` — controls SqlServer.Server build job -- `buildSqlClient` — controls MDS, Extensions.Azure, Abstractions, Logging (when AKV is disabled), and validation stages -- `buildAKVProvider` — controls AKV Provider build (also requires `buildSqlClient=true`) and Logging (when SqlClient is disabled) - -When set to `false`, the respective jobs/stages are excluded at compile-time using `${{ if }}` conditionals. This allows faster pipeline runs when only certain packages need to be built. - -### 5.2 Release Parameters - -Six boolean parameters control selective package release. All default to `false` so the release stage is skipped on normal CI/scheduled builds: - -```yaml -parameters: - - name: releaseSqlServerServer - displayName: 'Release Microsoft.SqlServer.Server' - type: boolean - default: false - - - name: releaseLogging - displayName: 'Release Microsoft.Data.SqlClient.Internal.Logging' - type: boolean - default: false - - - name: releaseAbstractions - displayName: 'Release Microsoft.Data.SqlClient.Extensions.Abstractions' - type: boolean - default: false - - - name: releaseSqlClient - displayName: 'Release Microsoft.Data.SqlClient' - type: boolean - default: false - - - name: releaseAzure - displayName: 'Release Microsoft.Data.SqlClient.Extensions.Azure' - type: boolean - default: false - - - name: releaseAKVProvider - displayName: 'Release Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider' - type: boolean - default: false -``` - -#### Dry-Run Parameter (Official Pipeline Only) - -The **official pipeline** includes a `releaseDryRun` parameter that defaults to `true` for safety: - -```yaml - - name: releaseDryRun - displayName: 'Release Dry Run (do not push to NuGet)' - type: boolean - default: true # safety default — must explicitly disable for real releases -``` - -When `releaseDryRun: true`, publish jobs download artifacts and list packages but skip actual NuGet push. Set to `false` for production releases. - -> **Note**: The non-official pipeline does **not** expose this parameter — dry-run mode is hardcoded and cannot be disabled. - ---- - -## 6. Variable & Version Management - -### 6.1 Variable Chain - -Variables are defined in a layered template chain. All variable groups live inside the templates — none are declared inline at the pipeline level: - -``` -dotnet-sqlclient-official-pipeline.yml - └─ libraries/variables.yml - └─ libraries/build-variables.yml - ├─ group: 'Release Variables' - ├─ group: 'Symbols publishing' ← SymbolsPublishServer, SymbolsPublishTokenUri, etc. - └─ libraries/common-variables.yml - ├─ group: 'ESRP Federated Creds (AME)' ← ESRP signing credentials - ├─ SymbolServer / SymbolTokenUri aliases ← mapped from Symbols publishing group - └─ all package versions, paths, build variables -``` - -### 6.2 Package Version Variables - -All package versions are centralized in `libraries/common-variables.yml`: - -| Package | GA Version Var | Preview Version Var | Assembly Version Var | -|---------|---------------|--------------------|--------------------| -| Logging | `loggingPackageVersion` | `loggingPackagePreviewVersion` | `loggingAssemblyFileVersion` | -| Abstractions | `abstractionsPackageVersion` | `abstractionsPackagePreviewVersion` | `abstractionsAssemblyFileVersion` | -| SqlServer.Server | `sqlServerPackageVersion` | `sqlServerPackagePreviewVersion` | `sqlServerAssemblyFileVersion` | -| SqlClient (MDS) | `mdsPackageVersion` | `mdsPackagePreviewVersion` | `mdsAssemblyFileVersion` | -| Extensions.Azure | `azurePackageVersion` | `azurePackagePreviewVersion` | `azureAssemblyFileVersion` | -| AKV Provider | `akvPackageVersion` | `akvPackagePreviewVersion` | `akvAssemblyFileVersion` | - -The pipeline resolves `effective*Version` variables at compile time based on the `isPreview` parameter. - -### 6.3 Release & Symbol Variables - -| Variable | Defined In | Purpose | -|----------|-----------|---------| -| `NuGetServiceConnection` | `libraries/common-variables.yml` | External NuGet service connection name for `NuGetCommand@2` push | -| `SymbolServer` | `libraries/common-variables.yml` (alias) | Alias for `$(SymbolsPublishServer)` — used by MDS `publish-symbols-step.yml` | -| `SymbolTokenUri` | `libraries/common-variables.yml` (alias) | Alias for `$(SymbolsPublishTokenUri)` — used by MDS `publish-symbols-step.yml` | - -### 6.4 Variable Groups - -| Group | Included In | Purpose | -|-------|------------|---------| -| `Release Variables` | `build-variables.yml` | Release-specific configuration | -| `Symbols publishing` | `build-variables.yml` | Symbol publishing credentials (`SymbolsAzureSubscription`, `SymbolsPublishServer`, `SymbolsPublishTokenUri`, `SymbolsUploadAccount`, `SymbolsPublishProjectName`) | -| `ESRP Federated Creds (AME)` | `common-variables.yml` | Federated identity for ESRP signing (`ESRPConnectedServiceName`, `ESRPClientId`, `AppRegistrationClientId`, `AppRegistrationTenantId`, `AuthAKVName`, `AuthSignCertName`) | - ---- - -## 7. Code Signing (ESRP) - -All packages are signed using **ESRP (Enterprise Security Release Pipeline)** with federated identity authentication. - -### Signing Flow (per job) - -#### csproj-based Extension Packages (Logging, Abstractions, Azure) -1. **Build DLLs only** — `build.proj` target (e.g., `BuildLogging`) compiles assemblies without creating NuGet packages -2. **ESRP DLL signing** — Assemblies are signed with Authenticode certificates via ESRP -3. **NuGet pack** — `build.proj` pack target (e.g., `PackLogging`) creates `.nupkg` from signed DLLs using `NoBuild=true` -4. **ESRP NuGet signing** — The `.nupkg` files are signed with NuGet certificates via ESRP - -This workflow ensures the NuGet package contains **signed DLLs** rather than signing the NuGet package around unsigned assemblies. - -#### nuspec-based Packages (SqlServer.Server, SqlClient, AKV Provider) -1. **Build + pack** — MSBuild creates both assemblies and NuGet packages -2. **ESRP DLL signing** — Assemblies are signed with Authenticode certificates via ESRP -3. **ESRP NuGet signing** — The `.nupkg` files are signed with NuGet certificates via ESRP - -### Credential Model - -- Extension packages (Logging, Abstractions, Azure, SqlServer.Server, SqlClient) use the primary ESRP credentials from the `ESRP Federated Creds (AME)` variable group (loaded via `common-variables.yml`) -- AKV Provider uses separate `Signing`-prefixed credential parameters that are passed explicitly to the `build-akv-official-job.yml` template -- All credentials are sourced from Azure Key Vault and federated identity — no secrets stored in pipeline YAML - ---- - -## 8. SDL & Compliance (OneBranch) - -Both pipelines use **OneBranch governed templates** for 1ES compliance. The SDL configuration differs between Official and Non-Official: - -| SDL Tool | Official | Non-Official | Purpose | -|----------|----------|--------------|---------| -| **TSA** | ✅ `enabled: true` | ❌ `enabled: false` | Uploads SDL results to TSA for downstream analysis | -| **ApiScan** | ✅ `enabled: true`, `break: true` | ✅ `enabled: true`, `break: true` | Scans APIs for compliance issues | -| **CodeQL** | ✅ (non-preview) | ✅ (non-preview) | Static analysis for security vulnerabilities | -| **SBOM** | ✅ (non-preview) | ✅ (non-preview) | Software Bill of Materials generation | -| **Policheck** | ✅ `break: true` | ✅ `break: true` | Scans for policy-violating content | -| **BinSkim** | ✅ (async, non-preview) | ✅ (async, non-preview) | Binary security analysis | -| **CredScan** | ✅ (async, non-preview) | ✅ (async, non-preview) | Credential leak detection | -| **Roslyn** | ✅ (async, non-preview) | ✅ (async, non-preview) | Roslyn-based security analyzers | -| **Armory** | ✅ `break: true` | ✅ `break: true` | Additional security scanning | - -### APIScan Configuration - -APIScan is configured at **both pipeline level and job level**: - -**Pipeline-level** (`globalSdl:apiscan:`): Sets default configuration inherited by all jobs. This is configured for MDS (Microsoft.Data.SqlClient) as the primary product. - -**Job-level** (`ob_sdl_apiscan_*` variables): Each build job overrides the pipeline defaults with package-specific settings: - -| Variable | Purpose | -|----------|---------| -| `ob_sdl_apiscan_enabled` | Enable/disable APIScan for this job (`true`) | -| `ob_sdl_apiscan_softwareFolder` | Path to signed DLLs for scanning | -| `ob_sdl_apiscan_symbolsFolder` | Path to PDBs for scanning | -| `ob_sdl_apiscan_softwarename` | Package name (e.g., `Microsoft.Data.SqlClient.Internal.Logging`) | -| `ob_sdl_apiscan_versionNumber` | Assembly file version | - -Each job copies its signed DLLs and PDBs to a package-specific folder under `$(Build.SourcesDirectory)/apiScan//` after ESRP DLL signing, ensuring APIScan analyzes the correct signed binaries for each package. - -> **PRC Compliance**: The Official pipeline hardcodes `OneBranch.Official.CrossPlat.yml` (not parameterized) to satisfy Production Readiness Check static verification requirements. - ---- - -## 9. Artifact Strategy - -- Each build job publishes its output as a **pipeline artifact** managed by OneBranch's `ob_outputDirectory` convention. -- Artifact names follow the OneBranch auto-generated pattern: `drop__` (e.g., `drop_build_dependent_build_package_SqlClient`). -- Downstream stages use `DownloadPipelineArtifact@2` to pull required packages into a local directory. -- A local NuGet source is configured at build time pointing to the downloaded artifacts directory so `dotnet restore` resolves internal dependencies. - ---- - -## 10. Trigger Configuration - -### Official Pipeline (`dotnet-sqlclient-official-pipeline.yml`) - -```yaml -trigger: - branches: - include: - - internal/main - paths: - include: - - .azuredevops - - .config - - doc - - eng/pipelines - - src - - tools - - azurepipelines-coverage.yml - - build.proj - - NuGet.config - -schedules: - - cron: '30 4 * * Mon' # Weekly Sunday 9:30 PM (UTC-7) - branches: { include: [internal/main] } - always: true - - cron: '30 3 * * Mon-Fri' # Weekday 8:30 PM (UTC-7) - branches: { include: [internal/main] } -``` - -- **CI trigger**: Runs on pushes to `internal/main` when relevant paths change -- **Scheduled**: Weekly full build (Sundays) + weekday builds (Mon–Fri) -- **No PR trigger**: Official pipeline should not run on PRs (separate PR pipelines exist) - -### Non-Official Pipeline (`dotnet-sqlclient-non-official-pipeline.yml`) - -```yaml -trigger: none -pr: none -``` - -- **Manual only**: Queued on-demand for validation/test builds - ---- - -## 11. Infrastructure - -| Concern | Implementation | -|---------|---------------| -| **Pipeline template** | OneBranch governed templates (`OneBranch.Pipelines/GovernedTemplates`) | -| **Build agents** | OneBranch-managed Windows containers (`WindowsHostVersion: 1ESWindows2022`) | -| **.NET SDK** | Pinned via `global.json` (with `useGlobalJson: true` in install steps) | -| **Code signing** | ESRP v2 with federated identity (Azure Key Vault backed) | -| **Symbol publishing** | Optional, controlled by `publishSymbols` parameter; uses `Symbols publishing` variable group (aliases `SymbolServer`/`SymbolTokenUri` defined in `common-variables.yml`) | - ---- - -## 12. Key Design Decisions - -1. **Single pipeline, multiple stages** — avoids managing 6 separate pipelines while keeping clear separation of concerns. -2. **Official + Non-Official variants** — hardcoded OneBranch templates (no parameterized `oneBranchType`) for PRC compliance; Non-Official variant allows manual validation builds. -3. **Parallel jobs within stages** — minimizes total wall-clock time; only waits where dependencies demand it. -4. **Pipeline artifacts over Universal Packages** — faster, ephemeral, scoped to the run; appropriate for build-time dependency resolution. -5. **ESRP-based code signing** — all DLLs and NuGet packages are signed in-pipeline using ESRP with federated identity; no secrets in YAML. -6. **Centralized version management** — all 6 package versions (GA + preview) defined once in `libraries/common-variables.yml`; `isPreview` toggle selects the active set. -7. **Dependency-aware stage ordering** — ensures packages are always built after their dependencies, guaranteeing consistent, reproducible builds. -8. **Validation in parallel with Stage 3** — MDS package validation runs alongside AKV Provider build (both depend on Stage 2), reducing total pipeline duration. -9. **Selective on-demand release** — 6 boolean parameters control which packages are published; the release stage is entirely skipped when none are selected, keeping normal CI builds unaffected. -10. **ADO Environment approval gate** — two environments: `NuGet-Production` (official, with configured approvals) and `NuGet-DryRun` (non-official, validation only). Both use `ob_release_environment` for OneBranch integration. -11. **Compile-time conditional publish jobs** — `${{ if eq(parameters.releaseXxx, true) }}` template expansion ensures unselected publish jobs are excluded entirely from the pipeline run (not just skipped at runtime). -12. **Mandatory dry-run for non-official** — the non-official variant hardcodes `dryRun: true` (no parameter), preventing accidental publication. The official variant defaults `releaseDryRun: true` for safety but allows override for actual releases. -13. **Selective build parameters** — `buildSqlClient`, `buildSqlServerServer`, and `buildAKVProvider` allow building subsets of packages, with dependency-aware conditionals ensuring Logging builds when either SqlClient or AKV is needed. +## Package Dependency Order + +Respect this graph when modifying build stages: + +1. `Microsoft.SqlServer.Server` — no dependencies +2. `Microsoft.Data.SqlClient.Internal.Logging` — no dependencies +3. `Microsoft.Data.SqlClient.Extensions.Abstractions` — depends on Logging +4. `Microsoft.Data.SqlClient` — depends on Logging + Abstractions +5. `Microsoft.Data.SqlClient.Extensions.Azure` — depends on Abstractions + Logging +6. `Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider` — depends on SqlClient + Abstractions + Logging + +## Build Stages + +Defined in `stages/build-stages.yml`. Four build stages plus validation, ordered by dependency: + +- **`build_independent`** (Stage 1) — Logging and SqlServer.Server in parallel; no inter-package dependencies +- **`build_abstractions`** (Stage 2) — Abstractions; `dependsOn: build_independent`; downloads Logging artifact +- **`build_dependent`** (Stage 3) — SqlClient and Extensions.Azure in parallel; `dependsOn: build_abstractions`; downloads Abstractions + Logging artifacts +- **`build_addons`** (Stage 4) — AKV Provider; `dependsOn: build_dependent`; downloads SqlClient + Abstractions + Logging artifacts +- **`mds_package_validation`** — Validates signed SqlClient package; `dependsOn: build_dependent`; runs in parallel with Stage 4 + +Stage conditional rules: +- Wrap stages/jobs in `${{ if }}` compile-time conditionals based on build parameters +- `buildSqlClient` controls Stages 2, 3, validation, and Logging (when AKV is disabled) +- `buildAKVProvider AND buildSqlClient` controls Stage 4 +- `buildSqlServerServer` controls SqlServer.Server job in Stage 1 +- Logging builds when `buildAKVProvider OR buildSqlClient` is true + +## Job Templates + +- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing +- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing +- **`validate-signed-package-job.yml`** — Validates signed MDS package (signature, strong names, folder structure, target frameworks) +- **`publish-nuget-package-job.yml`** — Reusable release job using OneBranch `templateContext.type: releaseJob` with `inputs` for artifact download; pushes via `NuGetCommand@2` + +When adding a new csproj-based package: +- Use `build-signed-csproj-package-job.yml` with appropriate `packageName`, `packageFullName`, `versionProperties`, and `downloadArtifacts` +- Add build and pack targets to `build.proj` +- Add version variables to `variables/common-variables.yml` +- Add artifact name variable to `variables/onebranch-variables.yml` + +## Release Stage + +- Defined in `stages/release-stages.yml`; produces stage `release_production` (official) or `release_test` (non-official) via `stageNameSuffix` parameter +- Entire stage excluded at compile time when no release parameters are true +- `dependsOn` is conditional based on which release parameters are set +- `releaseToProduction` parameter controls NuGet target feed: + - `true` → service connection `ADO Nuget Org Connection` (NuGet Production) + - `false` → service connection `ADO Nuget Org Test Connection` (NuGet Test) +- Non-official pipeline always sets `releaseToProduction: false` +- Environment gating: + - Official: `ob_release_environment: Production`, `ob_deploymentjob_environment: NuGet-Production` + - Non-official: `ob_release_environment: Test`, `ob_deploymentjob_environment: NuGet-DryRun` +- Each publish job uses OneBranch deployment job syntax (`templateContext.type: releaseJob` with `inputs` for artifact download) + +## Parameters + +Build parameters (all boolean, default `true`): +- `debug` — enable debug output (default `false`) +- `isPreview` — use preview version numbers (default `false`) +- `publishSymbols` — publish symbols to servers (default `false`) +- `buildSqlServerServer` — build SqlServer.Server package +- `buildSqlClient` — build SqlClient, Extensions.Azure, Abstractions, and Logging +- `buildAKVProvider` — build AKV Provider (requires `buildSqlClient`) + +Release parameters (all boolean, default `false`): +- `releaseSqlServerServer`, `releaseLogging`, `releaseAbstractions`, `releaseSqlClient`, `releaseAzure`, `releaseAKVProvider` + +Official-only parameter: +- `releaseToProduction` — push to NuGet Production feed (default `false`) + +When `isPreview` is true, pipeline resolves `effective*Version` variables to preview versions; otherwise GA versions. All versions defined in `variables/common-variables.yml`. + +## Variables and Versions + +- Variable chain: pipeline YAML → `variables/onebranch-variables.yml` → `variables/common-variables.yml` +- All package versions (GA, preview, assembly file) centralized in `variables/common-variables.yml` +- `effective*Version` pipeline variables map to selected version set based on `isPreview` +- Artifact name variables defined in `variables/onebranch-variables.yml` following `drop__` pattern +- `assemblyBuildNumber` derived from first segment of `Build.BuildNumber` only (16-bit limit) +- When adding a new package, add GA version, preview version, and assembly file version entries + +Variable groups: +- `Release Variables` — release configuration (in `common-variables.yml`) +- `Symbols publishing` — symbol publishing credentials (in `common-variables.yml`) +- `ESRP Federated Creds (AME)` — ESRP signing credentials (in `common-variables.yml`) + +## Code Signing (ESRP) + +- Uses ESRP v6 tasks (`EsrpMalwareScanning@6`, `EsrpCodeSigning@6`) with MSI/federated identity authentication +- Signing only runs when `isOfficial: true` — non-official pipelines skip ESRP steps +- csproj-based packages: sign DLLs first → pack with `NoBuild=true` → sign NuGet package (ensures NuGet contains signed DLLs) +- SqlClient: sign DLLs (including resource DLLs) → nuspec pack → sign NuGet package +- DLL signing uses keyCode `CP-230012` (Authenticode); NuGet signing uses keyCode `CP-401405` +- All ESRP credentials come from variable groups — never hardcode secrets in YAML + +## SDL and Compliance + +- TSA: enabled only in official pipeline; disabled in non-official to avoid spurious alerts +- ApiScan: enabled in both; currently `break: false` pending package registration +- Each build job sets `ob_sdl_apiscan_*` variables pointing to `$(Build.SourcesDirectory)/apiScan//` +- CodeQL, SBOM, Policheck (`break: true`): enabled in both pipelines +- asyncSdl `enabled: false` in both; individual sub-tools (CredScan, BinSkim, Armory, Roslyn) configured underneath +- Policheck exclusions: `$(REPO_ROOT)\.config\PolicheckExclusions.xml` +- CredScan suppressions: `$(REPO_ROOT)/.config/CredScanSuppressions.json` + +## Artifact Conventions + +- `ob_outputDirectory` set to `$(PACK_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory +- Artifact names follow `drop__` — defined in `variables/onebranch-variables.yml` +- Downstream jobs download artifacts via `DownloadPipelineArtifact@2` into `$(Build.SourcesDirectory)/packages` +- Downloaded packages serve as a local NuGet source for `dotnet restore` +- If stage or job names change, update artifact name variables in `onebranch-variables.yml` + +## Common Pitfalls + +- Do not use `PublishPipelineArtifacts` task — OneBranch auto-publishes from `ob_outputDirectory` +- Do not add `NuGetToolInstaller@1` in OneBranch containers — NuGet is pre-installed +- Variable templates are under `variables/` not `libraries/` +- Always test parameter changes in the non-official pipeline first +- When modifying stage names, update all `dependsOn` references and artifact name variables +- Release jobs must use `templateContext.type: releaseJob` with `inputs` for artifact download — deployment jobs do not auto-download artifacts diff --git a/eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml b/eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml index 809690a886..e257386918 100644 --- a/eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml +++ b/eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml @@ -70,6 +70,11 @@ parameters: - name: publishSymbols type: boolean + # True to enable ESRP malware scanning and code signing steps, which should not be + # run on non-official pipelines as they access production resources. + - name: isOfficial + type: boolean + # Values required by ESRP tasks. - name: esrpConnectedServiceName type: string @@ -144,18 +149,19 @@ jobs: buildConfiguration: ${{ parameters.buildConfiguration }} versionProperties: ${{ parameters.versionProperties }} - # ESRP sign the DLLs. - - template: /eng/pipelines/onebranch/steps/compound-esrp-dll-signing-step.yml@self - parameters: - appRegistrationClientId: ${{ parameters.appRegistrationClientId }} - appRegistrationTenantId: ${{ parameters.appRegistrationTenantId }} - authAkvName: ${{ parameters.authAkvName }} - authSignCertName: ${{ parameters.authSignCertName }} - esrpClientId: ${{ parameters.esrpClientId }} - esrpConnectedServiceName: ${{ parameters.esrpConnectedServiceName }} - pattern: ${{ parameters.packageFullName }}.dll - - # Copy signed DLLs and PDBs to APIScan folders. + - ${{ if eq(parameters.isOfficial, true) }}: + # ESRP sign the DLLs. + - template: /eng/pipelines/onebranch/steps/compound-esrp-dll-signing-step.yml@self + parameters: + appRegistrationClientId: ${{ parameters.appRegistrationClientId }} + appRegistrationTenantId: ${{ parameters.appRegistrationTenantId }} + authAkvName: ${{ parameters.authAkvName }} + authSignCertName: ${{ parameters.authSignCertName }} + esrpClientId: ${{ parameters.esrpClientId }} + esrpConnectedServiceName: ${{ parameters.esrpConnectedServiceName }} + pattern: ${{ parameters.packageFullName }}.dll + + # Copy signed/unsigned DLLs and PDBs to APIScan folders. - task: CopyFiles@2 displayName: Copy DLLs for APIScan inputs: @@ -182,15 +188,17 @@ jobs: buildConfiguration: ${{ parameters.buildConfiguration }} versionProperties: ${{ parameters.versionProperties }} - # ESRP sign the NuGet package. - - template: /eng/pipelines/onebranch/steps/compound-esrp-nuget-signing-step.yml@self - parameters: - appRegistrationClientId: ${{ parameters.appRegistrationClientId }} - appRegistrationTenantId: ${{ parameters.appRegistrationTenantId }} - authAkvName: ${{ parameters.authAkvName }} - authSignCertName: ${{ parameters.authSignCertName }} - esrpClientId: ${{ parameters.esrpClientId }} - esrpConnectedServiceName: ${{ parameters.esrpConnectedServiceName }} + - ${{ if eq(parameters.isOfficial, true) }}: + # ESRP sign the NuGet package. + - template: /eng/pipelines/onebranch/steps/compound-esrp-nuget-signing-step.yml@self + parameters: + appRegistrationClientId: ${{ parameters.appRegistrationClientId }} + appRegistrationTenantId: ${{ parameters.appRegistrationTenantId }} + authAkvName: ${{ parameters.authAkvName }} + authSignCertName: ${{ parameters.authSignCertName }} + esrpClientId: ${{ parameters.esrpClientId }} + esrpConnectedServiceName: ${{ parameters.esrpConnectedServiceName }} + pattern: ${{ parameters.packageFullName }}.*nupkg # Publish symbols to servers - ${{ if eq(parameters.publishSymbols, true) }}: diff --git a/eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml b/eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml index a422a5351c..f3033b07f0 100644 --- a/eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml +++ b/eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml @@ -16,122 +16,130 @@ parameters: - name: isPreview type: boolean + # True to enable ESRP malware scanning and code signing steps, which should not + # be run on non-official pipelines as they access production resources. + - name: isOfficial + type: boolean + jobs: -- job: build_package_SqlClient - displayName: 'Build Microsoft.Data.SqlClient' - pool: - type: windows # read more about custom job pool types at https://aka.ms/obpipelines/yaml/jobs - - variables: - ob_outputDirectory: $(PACK_OUTPUT) - # APIScan configuration for this Extension package - ob_sdl_apiscan_enabled: true - ob_sdl_apiscan_softwareFolder: $(Build.SourcesDirectory)/apiScan/SqlClient/dlls - ob_sdl_apiscan_symbolsFolder: $(Build.SourcesDirectory)/apiScan/SqlClient/pdbs - ob_sdl_apiscan_softwarename: Microsoft.Data.SqlClient - ob_sdl_apiscan_versionNumber: $(assemblyBuildNumber) - - ${{ if parameters.isPreview }}: - abstractionsPackageVersion: $(abstractionsPackagePreviewVersion) - loggingPackageVersion: $(loggingPackagePreviewVersion) - mdsPackageVersion: $(mdsPackagePreviewVersion) - - steps: - - script: SET - displayName: 'Print Environment Variables' - - # Download the Abstractions and Logging packages from the previous stage into - # packages/ so that they're available via the local NuGet feed when restoring MDS. - # MDS depends on both Extensions.Abstractions and Internal.Logging. - - task: DownloadPipelineArtifact@2 - displayName: Download Abstractions Package - inputs: - artifactName: $(abstractionsArtifactsName) - targetPath: $(Build.SourcesDirectory)/packages - - - task: DownloadPipelineArtifact@2 - displayName: Download Logging Package - inputs: - artifactName: $(loggingArtifactsName) - targetPath: $(Build.SourcesDirectory)/packages - - # Install the .NET SDK. - - template: /eng/pipelines/steps/install-dotnet.yml@self - - # Build our tooling, which is required by the analysis step below, but - # shouldn't be analyzed itself. - - task: MSBuild@1 - displayName: 'Build Tooling' - inputs: - solution: '**/build.proj' - configuration: Release - msbuildArguments: -t:BuildTools - - # Perform analysis before building, since this step will clobber build output. - - template: /eng/pipelines/onebranch/steps/code-analyze-step.yml@self - - # Build MDS, producing signed DLLs. - - template: /eng/pipelines/onebranch/steps/build-all-configurations-signed-dlls-step.yml@self - parameters: - # These variables are sourced from common-variables.yml. - abstractionsAssemblyFileVersion: $(abstractionsAssemblyFileVersion) - abstractionsPackageVersion: $(abstractionsPackageVersion) - loggingAssemblyFileVersion: $(loggingAssemblyFileVersion) - loggingPackageVersion: $(loggingPackageVersion) - mdsAssemblyFileVersion: $(mdsAssemblyFileVersion) - mdsPackageVersion: $(mdsPackageVersion) - - - template: /eng/pipelines/onebranch/steps/esrp-code-signing-step.yml@self - parameters: - artifactType: dll - sourceRoot: $(BUILD_OUTPUT) - dllPattern: 'Microsoft.Data.SqlClient.dll' - - - template: /eng/pipelines/onebranch/steps/esrp-code-signing-step.yml@self - parameters: - artifactType: dll - sourceRoot: $(BUILD_OUTPUT) - dllPattern: 'Microsoft.Data.SqlClient.resources.dll' - - - template: /eng/pipelines/common/templates/steps/generate-nuget-package-step.yml@self - parameters: - buildConfiguration: Release - displayName: 'Create MDS NuGet Package' - generateSymbolsPackage: true - installNuget: false - nuspecPath: $(nuspecPath) - outputDirectory: $(PACK_OUTPUT) - packageVersion: $(mdsPackageVersion) - properties: 'AbstractionsPackageVersion=$(abstractionsPackageVersion);LoggingPackageVersion=$(loggingPackageVersion)' - referenceType: Package - - - template: /eng/pipelines/onebranch/steps/esrp-code-signing-step.yml@self - parameters: - artifactType: pkg - - # Copy signed DLLs and PDBs to APIScan folders. - - task: CopyFiles@2 - displayName: Copy DLLs for APIScan - inputs: - SourceFolder: $(BUILD_OUTPUT)/Package/bin - Contents: '**/Microsoft.Data.SqlClient.dll' - TargetFolder: $(ob_sdl_apiscan_softwareFolder) - # We must preserve the folder structure since our C# projects may produce multiple - # identically named DLLs for different target frameworks (e.g. netstandard2.0, net5.0, - # etc.), and we need to keep those separate for APIScan to work correctly. - flattenFolders: false - - - task: CopyFiles@2 - displayName: Copy PDBs for APIScan - inputs: - SourceFolder: $(BUILD_OUTPUT)/Package/bin - Contents: '**/Microsoft.Data.SqlClient.pdb' - TargetFolder: $(ob_sdl_apiscan_symbolsFolder) - flattenFolders: false - - # Publish symbols to servers - - ${{ if eq(parameters.publishSymbols, true) }}: - - template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self - parameters: - packageFullName: Microsoft.Data.SqlClient - packageVersion: $(mdsPackageVersion) + - job: build_package_SqlClient + displayName: "Build Microsoft.Data.SqlClient" + pool: + type: windows # read more about custom job pool types at https://aka.ms/obpipelines/yaml/jobs + + variables: + ob_outputDirectory: $(PACK_OUTPUT) + # APIScan configuration for this Extension package + ob_sdl_apiscan_enabled: true + ob_sdl_apiscan_softwareFolder: $(Build.SourcesDirectory)/apiScan/SqlClient/dlls + ob_sdl_apiscan_symbolsFolder: $(Build.SourcesDirectory)/apiScan/SqlClient/pdbs + ob_sdl_apiscan_softwarename: Microsoft.Data.SqlClient + ob_sdl_apiscan_versionNumber: $(assemblyBuildNumber) + + ${{ if parameters.isPreview }}: + abstractionsPackageVersion: $(abstractionsPackagePreviewVersion) + loggingPackageVersion: $(loggingPackagePreviewVersion) + mdsPackageVersion: $(mdsPackagePreviewVersion) + + steps: + - script: SET + displayName: "Print Environment Variables" + + # Download the Abstractions and Logging packages from the previous stage into + # packages/ so that they're available via the local NuGet feed when restoring MDS. + # MDS depends on both Extensions.Abstractions and Internal.Logging. + - task: DownloadPipelineArtifact@2 + displayName: Download Abstractions Package + inputs: + artifactName: $(abstractionsArtifactsName) + targetPath: $(Build.SourcesDirectory)/packages + + - task: DownloadPipelineArtifact@2 + displayName: Download Logging Package + inputs: + artifactName: $(loggingArtifactsName) + targetPath: $(Build.SourcesDirectory)/packages + + # Install the .NET SDK. + - template: /eng/pipelines/steps/install-dotnet.yml@self + + # Build our tooling, which is required by the analysis step below, but + # shouldn't be analyzed itself. + - task: MSBuild@1 + displayName: "Build Tooling" + inputs: + solution: "**/build.proj" + configuration: Release + msbuildArguments: -t:BuildTools + + # Perform analysis before building, since this step will clobber build output. + - template: /eng/pipelines/onebranch/steps/code-analyze-step.yml@self + + # Build MDS, producing signed DLLs. + - template: /eng/pipelines/onebranch/steps/build-all-configurations-signed-dlls-step.yml@self + parameters: + # These variables are sourced from common-variables.yml. + abstractionsAssemblyFileVersion: $(abstractionsAssemblyFileVersion) + abstractionsPackageVersion: $(abstractionsPackageVersion) + loggingAssemblyFileVersion: $(loggingAssemblyFileVersion) + loggingPackageVersion: $(loggingPackageVersion) + mdsAssemblyFileVersion: $(mdsAssemblyFileVersion) + mdsPackageVersion: $(mdsPackageVersion) + + - ${{ if eq(parameters.isOfficial, true) }}: + - template: /eng/pipelines/onebranch/steps/esrp-code-signing-step.yml@self + parameters: + artifactType: dll + sourceRoot: $(BUILD_OUTPUT) + dllPattern: "Microsoft.Data.SqlClient.dll" + + - template: /eng/pipelines/onebranch/steps/esrp-code-signing-step.yml@self + parameters: + artifactType: dll + sourceRoot: $(BUILD_OUTPUT) + dllPattern: "Microsoft.Data.SqlClient.resources.dll" + + - template: /eng/pipelines/common/templates/steps/generate-nuget-package-step.yml@self + parameters: + buildConfiguration: Release + displayName: "Create MDS NuGet Package" + generateSymbolsPackage: true + installNuget: false + nuspecPath: $(nuspecPath) + outputDirectory: $(PACK_OUTPUT) + packageVersion: $(mdsPackageVersion) + properties: "AbstractionsPackageVersion=$(abstractionsPackageVersion);LoggingPackageVersion=$(loggingPackageVersion)" + referenceType: Package + + - ${{ if eq(parameters.isOfficial, true) }}: + - template: /eng/pipelines/onebranch/steps/esrp-code-signing-step.yml@self + parameters: + artifactType: pkg + nupkgPattern: "Microsoft.Data.SqlClient.$(mdsPackageVersion).*nupkg" + + # Copy signed DLLs and PDBs to APIScan folders. + - task: CopyFiles@2 + displayName: Copy DLLs for APIScan + inputs: + SourceFolder: $(BUILD_OUTPUT)/Package/bin + Contents: "**/Microsoft.Data.SqlClient.dll" + TargetFolder: $(ob_sdl_apiscan_softwareFolder) + # We must preserve the folder structure since our C# projects may produce multiple + # identically named DLLs for different target frameworks (e.g. netstandard2.0, net5.0, + # etc.), and we need to keep those separate for APIScan to work correctly. + flattenFolders: false + + - task: CopyFiles@2 + displayName: Copy PDBs for APIScan + inputs: + SourceFolder: $(BUILD_OUTPUT)/Package/bin + Contents: "**/Microsoft.Data.SqlClient.pdb" + TargetFolder: $(ob_sdl_apiscan_symbolsFolder) + flattenFolders: false + + # Publish symbols to servers + - ${{ if eq(parameters.publishSymbols, true) }}: + - template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self + parameters: + packageFullName: Microsoft.Data.SqlClient + packageVersion: $(mdsPackageVersion) diff --git a/eng/pipelines/onebranch/jobs/validate-signed-package-job.yml b/eng/pipelines/onebranch/jobs/validate-signed-package-job.yml index 656ea8f181..3b463d0b81 100644 --- a/eng/pipelines/onebranch/jobs/validate-signed-package-job.yml +++ b/eng/pipelines/onebranch/jobs/validate-signed-package-job.yml @@ -4,7 +4,6 @@ # See the LICENSE file in the project root for more information. # ################################################################################# parameters: - # The name of the pipeline artifacts to download prior to building the tests. - name: artifactName type: string @@ -13,269 +12,277 @@ parameters: - name: isPreview type: boolean + # True if this build is an official build. This will be used to gate some checks + # that only apply to official builds, such as signature verification. + - name: isOfficial + type: boolean + jobs: -- job: validate_signed_package - displayName: 'Verify signed package' - - pool: - type: windows # read more about custom job pool types at https://aka.ms/obpipelines/yaml/jobs - isCustom: true - name: ADO-1ES-Pool - vmImage: 'ADO-MMS22-SQL19' - - variables: # More settings at https://aka.ms/obpipelines/yaml/jobs - - template: /eng/pipelines/onebranch/variables/sqlclient-validation-variables.yml@self - - - name: pathToDownloadedNuget # path to the downloaded nuget files - value: $(Pipeline.Workspace)\${{parameters.artifactName }} - - - ${{ if parameters.isPreview }}: - - name: extractedNugetPath - value: $(extractedNugetRootPath).$(mdsPackagePreviewVersion) - - name: mdsPackageVersion - value: $(mdsPackagePreviewVersion) - - steps: - - script: SET - displayName: 'Print Environment Variables' - - - task: NuGetToolInstaller@1 - displayName: 'Use NuGet' - - - powershell: | - # Displays the paths of all the local cache directories - nuget locals all -List - - #Clears all files from all local cache directories - nuget locals all -Clear - displayName: 'Clear local cache' - - - download: current - artifact: ${{parameters.artifactName}} - patterns: '**/*.*nupkg' - displayName: 'Download NuGet Package' - - - powershell: | - # Install nuget package - Install-Package -Name "Microsoft.Data.SqlClient" -Destination "$(TempFolderName)" -Force -Source $(pathToDownloadedNuget) -SkipDependencies - - Write-Host "--------------------------------------------------" - Write-Host '$(TempFolderName)' - ls $(TempFolderName) - Write-Host "--------------------------------------------------" - displayName: 'Extract Nuget in temp folder' - - - powershell: | - Write-Host "--------------------------------------------------" - Write-Host "This will verify the artifact signature" -ForegroundColor Green - Write-Host "--------------------------------------------------" - - nuget verify -All $(pathToDownloadedNuget)\*.nupkg - nuget verify -All $(pathToDownloadedNuget)\*.snupkg - displayName: 'Verify nuget signature' - - - powershell: | - # Recursively find all .dll files in TempFolder (installed nuget folder) - # Microsoft.Data.SqlClient.dll and Microsoft.Data.SqlClient.resources.dll (in localized folders) should have strong name - $dllFiles = Get-ChildItem -Path $(TempFolderName) -Recurse -Filter *.dll - $badDlls = @() - foreach ($file in $dllFiles) - { - # Run sn.exe to verify the strong name on each dll - $result = & "C:\Program Files (x86)\Microsoft SDKs\Windows\*\bin\NETFX 4.8.1 Tools\sn.exe" -vf $file.FullName - Write-OutPut $result - - # if the dll is not valid, it would be delay signed or test-signed which is not meant for production - if($result[$result.Length-1] -notlike "* is valid") - { - $badDlls += $result[$result.Length-1] - } - } - if($badDlls.Count -gt 0) - { - Write-OutPut "Error: Invalid dlls are detected. Check the list below:" - foreach($dll in $badDlls) - { - Write-Output $dll - } - Exit -1 - } - Write-Host "Strong name has been verified for all dlls" - displayName: 'Verify assembly strong names' - - - powershell: | - # Checks the expected folder names such as lib, ref, runtimes - Get-ChildItem -Path $(extractedNugetPath) -Directory | select Name | foreach { - if('$(expectedFolderNames)'.contains($_.Name)){ - Write-Host expected folder name verfied: $_.Name - } - } - displayName: 'Check expected folder names' - - - powershell: | - # Checks the version of DotNetFramework and DotNet - $countErr = 0 - $countPass = 0 - $excludNamesFromRuntimeFolder = 'lib','win','unix' - - Get-ChildItem -Path $(extractedNugetPath) -Directory | foreach { - $parentname=$_.Name - Write-Host $_.FullName -ForegroundColor yellow - - if($_.Name -ne 'runtimes') { - Get-ChildItem -Path $_.FullName -Directory | select Name | foreach { - if('$(expectedDotnetVersions)'.Contains($_.Name)){ - Write-Host "`tExpected version verified in $parentname": $_.Name -ForegroundColor green - $countPass += 1 + - job: validate_nuget_package + displayName: "Validate NuGet package" + + pool: + type: windows # read more about custom job pool types at https://aka.ms/obpipelines/yaml/jobs + isCustom: true + name: ADO-1ES-Pool + vmImage: "ADO-MMS22-SQL19" + + variables: # More settings at https://aka.ms/obpipelines/yaml/jobs + - template: /eng/pipelines/onebranch/variables/sqlclient-validation-variables.yml@self + + - name: pathToDownloadedNuget # path to the downloaded nuget files + value: $(Pipeline.Workspace)\${{parameters.artifactName }} + + - ${{ if parameters.isPreview }}: + - name: extractedNugetPath + value: $(extractedNugetRootPath).$(mdsPackagePreviewVersion) + - name: mdsPackageVersion + value: $(mdsPackagePreviewVersion) + + steps: + - script: SET + displayName: "Print Environment Variables" + + - task: NuGetToolInstaller@1 + displayName: "Use NuGet" + + - powershell: | + # Displays the paths of all the local cache directories + nuget locals all -List + + #Clears all files from all local cache directories + nuget locals all -Clear + displayName: "Clear local cache" + + - download: current + artifact: ${{parameters.artifactName}} + patterns: "**/*.*nupkg" + displayName: "Download NuGet Package" + + - powershell: | + # Install nuget package + Install-Package -Name "Microsoft.Data.SqlClient" -Destination "$(TempFolderName)" -Force -Source $(pathToDownloadedNuget) -SkipDependencies + + Write-Host "--------------------------------------------------" + Write-Host '$(TempFolderName)' + ls $(TempFolderName) + Write-Host "--------------------------------------------------" + displayName: "Extract Nuget in temp folder" + + - ${{ if eq(parameters.isOfficial, true) }}: + - powershell: | + Write-Host "--------------------------------------------------" + Write-Host "This will verify the artifact signature" -ForegroundColor Green + Write-Host "--------------------------------------------------" + + nuget verify -All $(pathToDownloadedNuget)\*.nupkg + nuget verify -All $(pathToDownloadedNuget)\*.snupkg + displayName: "Verify nuget signature" + + - powershell: | + # Recursively find all .dll files in TempFolder (installed nuget folder) + # Microsoft.Data.SqlClient.dll and Microsoft.Data.SqlClient.resources.dll (in localized folders) should have strong name + $dllFiles = Get-ChildItem -Path $(TempFolderName) -Recurse -Filter *.dll + $badDlls = @() + foreach ($file in $dllFiles) + { + # Run sn.exe to verify the strong name on each dll + $result = & "C:\Program Files (x86)\Microsoft SDKs\Windows\*\bin\NETFX 4.8.1 Tools\sn.exe" -vf $file.FullName + Write-OutPut $result + + # if the dll is not valid, it would be delay signed or test-signed which is not meant for production + if($result[$result.Length-1] -notlike "* is valid") + { + $badDlls += $result[$result.Length-1] } - else{ - Write-Host "`tUnexpected version detected in $parentname": $_.Name - $countErr += 1 + } + if($badDlls.Count -gt 0) + { + Write-OutPut "Error: Invalid dlls are detected. Check the list below:" + foreach($dll in $badDlls) + { + Write-Output $dll } + Exit -1 + } + Write-Host "Strong name has been verified for all dlls" + displayName: "Verify assembly strong names" + + - powershell: | + # Checks the expected folder names such as lib, ref, runtimes + Get-ChildItem -Path $(extractedNugetPath) -Directory | select Name | foreach { + if('$(expectedFolderNames)'.contains($_.Name)){ + Write-Host expected folder name verfied: $_.Name + } } - } + displayName: "Check expected folder names" + + - powershell: | + # Checks the version of DotNetFramework and DotNet + $countErr = 0 + $countPass = 0 + $excludNamesFromRuntimeFolder = 'lib','win','unix' + + Get-ChildItem -Path $(extractedNugetPath) -Directory | foreach { + $parentname=$_.Name + Write-Host $_.FullName -ForegroundColor yellow + + if($_.Name -ne 'runtimes') { + Get-ChildItem -Path $_.FullName -Directory | select Name | foreach { + if('$(expectedDotnetVersions)'.Contains($_.Name)){ + Write-Host "`tExpected version verified in $parentname": $_.Name -ForegroundColor green + $countPass += 1 + } + else{ + Write-Host "`tUnexpected version detected in $parentname": $_.Name + $countErr += 1 + } + } + } - elseif ($_.Name -eq 'runtimes'){ - Get-ChildItem -Depth 3 -Path $_.FullName -Exclude $excludNamesFromRuntimeFolder -Directory | foreach{ - if('$(expectedDotnetVersions)'.Contains($_.Name)){ - Write-Host "`tExpected version verfied in $parentname": $_.Name - $countPass += 1 + elseif ($_.Name -eq 'runtimes'){ + Get-ChildItem -Depth 3 -Path $_.FullName -Exclude $excludNamesFromRuntimeFolder -Directory | foreach{ + if('$(expectedDotnetVersions)'.Contains($_.Name)){ + Write-Host "`tExpected version verfied in $parentname": $_.Name + $countPass += 1 + } + else{ + Write-Host "`tUnexpected version detected": $_.Name -ForegroundColor Red + $countErr += 1 + } + } } else{ - Write-Host "`tUnexpected version detected": $_.Name -ForegroundColor Red - $countErr += 1 + Write-Host "`tUnknown folder " $_.Name -ForegroundColor Red + Exit -1 } } - } - else{ - Write-Host "`tUnknown folder " $_.Name -ForegroundColor Red - Exit -1 - } - } - - Write-Host "_______________" - Write-Host "Expected: $countPass" - Write-Host "Unexpected: $countErr" - Write-Host "_______________" - if ($countErr -ne 0) - { - Write-Host "Unexpected versions are detected!" -ForegroundColor Red - Exit -1 - } - displayName: 'Check Expected framework' - - - powershell: | - # list all the child items of created temp folder - - #Verify all DLLs unzipped match "expected" hierarchy - - foreach( $folderName in (Get-ChildItem -Path $(extractedNugetPath) -Directory).Name) - { - # List all Childerns of the Path - Get-ChildItem -Path $(extractedNugetPath)\$folderName -Recurse -File - $subFiles = Get-ChildItem -Path $(extractedNugetPath)\$folderName -Recurse -File - - foreach($file in $subFiles) - { - if($subFiles[0].Name -like "*.dll" ) + + Write-Host "_______________" + Write-Host "Expected: $countPass" + Write-Host "Unexpected: $countErr" + Write-Host "_______________" + if ($countErr -ne 0) { - Write-Host $subFiles[0].Name -ForegroundColor Green - Write-Host $subFiles[1].Name -ForegroundColor Green - if(($folderName -eq 'lib') -or ($folderName -eq 'ref')) + Write-Host "Unexpected versions are detected!" -ForegroundColor Red + Exit -1 + } + displayName: "Check Expected framework" + + - powershell: | + # list all the child items of created temp folder + + #Verify all DLLs unzipped match "expected" hierarchy + + foreach( $folderName in (Get-ChildItem -Path $(extractedNugetPath) -Directory).Name) + { + # List all Childerns of the Path + Get-ChildItem -Path $(extractedNugetPath)\$folderName -Recurse -File + $subFiles = Get-ChildItem -Path $(extractedNugetPath)\$folderName -Recurse -File + + foreach($file in $subFiles) { - if($subFiles[2].Name -like "*.dll") + if($subFiles[0].Name -like "*.dll" ) { - Write-Host $subFiles[2].Name -ForegroundColor Green + Write-Host $subFiles[0].Name -ForegroundColor Green + Write-Host $subFiles[1].Name -ForegroundColor Green + if(($folderName -eq 'lib') -or ($folderName -eq 'ref')) + { + if($subFiles[2].Name -like "*.dll") + { + Write-Host $subFiles[2].Name -ForegroundColor Green + } + else + { + $subFiles[2].Name + Write-Host "Expected file pattern for localization did not match to *.dll" -ForegroundColor Red + Exit -1 + } + } } else { - $subFiles[2].Name - Write-Host "Expected file pattern for localization did not match to *.dll" -ForegroundColor Red + $subFiles[0].Name + $subFiles[1].Name + Write-Host "Expected file pattern did not match to *.dll" -ForegroundColor Red Exit -1 } } } - else - { - $subFiles[0].Name - $subFiles[1].Name - Write-Host "Expected file pattern did not match to *.dll" -ForegroundColor Red - Exit -1 - } - } - } - displayName: 'Verify all DLLs unzipped match "expected" hierarchy' - - powershell: | - # Verify all dlls status are Valid - - $dlls = Get-ChildItem -Path $(extractedNugetPath) -Recurse -Include *.dll - foreach ($status in $dlls | Get-AuthenticodeSignature) - { - if ($status.Status -eq "Valid") + displayName: 'Verify all DLLs unzipped match "expected" hierarchy' + + - ${{ if eq(parameters.isOfficial, true) }}: + - powershell: | + # Verify all dlls status are Valid + + $dlls = Get-ChildItem -Path $(extractedNugetPath) -Recurse -Include *.dll + foreach ($status in $dlls | Get-AuthenticodeSignature) + { + if ($status.Status -eq "Valid") + { + Write-Host $status.Status $status.Path + } + else + { + Write-Host "dll status of '$status.Path' is not valid!" -ForegroundColor Red + $status + Exit -1 + } + } + displayName: "Verify all dlls status are Valid" + + - powershell: | + # This will check each DLL's ProductVersion and FileVersion against + # expected values. + $failed = 0 + + foreach ( $pVersion in Get-ChildItem *.dll -Path $(extractedNugetPath) -Recurse | ForEach-Object versioninfo ) { - Write-Host $status.Status $status.Path + if ($pVersion.ProductVersion -Like '$(mdsPackageVersion)*') + { + Write-Host -ForegroundColor Green "Correct ProductVersion detected for $($pVersion.FileName): $($pVersion.ProductVersion)" + } + else + { + Write-Host -ForegroundColor Red "Wrong ProductVersion detected for $($pVersion.FileName); expected: $(mdsPackageVersion); found: $($pVersion.ProductVersion)" + $failed = 1 + } + + if ($pVersion.FileVersion -eq '$(mdsAssemblyFileVersion)') + { + Write-Host -ForegroundColor Green "Correct FileVersion detected for $($pVersion.FileName): $($pVersion.FileVersion)" + } + else + { + Write-Host -ForegroundColor Red "Wrong FileVersion detected for $($pVersion.FileName); expected $(mdsAssemblyFileVersion); found: $($pVersion.FileVersion)" + $failed = 1 + } } - else + + if ($failed -ne 0) { - Write-Host "dll status of '$status.Path' is not valid!" -ForegroundColor Red - $status Exit -1 } - } - displayName: 'Verify all dlls status are Valid' - - powershell: | - # This will check each DLL's ProductVersion and FileVersion against - # expected values. - $failed = 0 + Get-ChildItem *.dll -Path $(extractedNugetPath) -Recurse | ForEach-Object VersionInfo | Format-List + displayName: 'Verify "File Version" matches expected values for DLLs' - foreach ( $pVersion in Get-ChildItem *.dll -Path $(extractedNugetPath) -Recurse | ForEach-Object versioninfo ) - { - if ($pVersion.ProductVersion -Like '$(mdsPackageVersion)*') - { - Write-Host -ForegroundColor Green "Correct ProductVersion detected for $($pVersion.FileName): $($pVersion.ProductVersion)" - } - else - { - Write-Host -ForegroundColor Red "Wrong ProductVersion detected for $($pVersion.FileName); expected: $(mdsPackageVersion); found: $($pVersion.ProductVersion)" - $failed = 1 - } + - powershell: | + # Check assembly versions. + # + # GOTCHA: This expects the Versions.props file having XML elements in a + # certain order. If the order changes, this check will fail! + # + # TODO: This also isn't checking the versions of the actual assemblies in + # the package, so it isn't terribly useful. - if ($pVersion.FileVersion -eq '$(mdsAssemblyFileVersion)') - { - Write-Host -ForegroundColor Green "Correct FileVersion detected for $($pVersion.FileName): $($pVersion.FileVersion)" - } - else + [Xml] $versionprops = Get-Content -Path "tools/props/Versions.props" + $AssemblyFileVersion = $versionprops.Project.PropertyGroup[2].AssemblyFileVersion + $AssemblyVersion = $versionprops.Project.PropertyGroup[2].AssemblyVersion + + if($AssemblyFileVersion -eq $AssemblyVersion) { - Write-Host -ForegroundColor Red "Wrong FileVersion detected for $($pVersion.FileName); expected $(mdsAssemblyFileVersion); found: $($pVersion.FileVersion)" - $failed = 1 + Write-Host AssemblyFileVersion: $AssemblyFileVersion should not be equal to: $AssemblyVersion + Exit -1 } - } - - if ($failed -ne 0) - { - Exit -1 - } - - Get-ChildItem *.dll -Path $(extractedNugetPath) -Recurse | ForEach-Object VersionInfo | Format-List - displayName: 'Verify "File Version" matches expected values for DLLs' - - - powershell: | - # Check assembly versions. - # - # GOTCHA: This expects the Versions.props file having XML elements in a - # certain order. If the order changes, this check will fail! - # - # TODO: This also isn't checking the versions of the actual assemblies in - # the package, so it isn't terribly useful. - - [Xml] $versionprops = Get-Content -Path "tools/props/Versions.props" - $AssemblyFileVersion = $versionprops.Project.PropertyGroup[2].AssemblyFileVersion - $AssemblyVersion = $versionprops.Project.PropertyGroup[2].AssemblyVersion - - if($AssemblyFileVersion -eq $AssemblyVersion) - { - Write-Host AssemblyFileVersion: $AssemblyFileVersion should not be equal to: $AssemblyVersion - Exit -1 - } - displayName: 'Check "AssemblyFileVersion" is not same as "AssemblyVersion" in version.props' + displayName: 'Check "AssemblyFileVersion" is not same as "AssemblyVersion" in version.props' diff --git a/eng/pipelines/onebranch/sqlclient-non-official.yml b/eng/pipelines/onebranch/sqlclient-non-official.yml index 2274d8cccf..25dbe98097 100644 --- a/eng/pipelines/onebranch/sqlclient-non-official.yml +++ b/eng/pipelines/onebranch/sqlclient-non-official.yml @@ -94,31 +94,31 @@ variables: # Define the effective versions for all of the packages we build and release. - ${{ if parameters.isPreview }}: - - name: effectiveSqlServerVersion - value: $(sqlServerPackagePreviewVersion) - - name: effectiveLoggingVersion - value: $(loggingPackagePreviewVersion) - - name: effectiveAbstractionsVersion - value: $(abstractionsPackagePreviewVersion) - - name: effectiveSqlClientVersion - value: $(mdsPackagePreviewVersion) - - name: effectiveAzureVersion - value: $(azurePackagePreviewVersion) - - name: effectiveAkvProviderVersion - value: $(akvPackagePreviewVersion) + - name: effectiveSqlServerVersion + value: $(sqlServerPackagePreviewVersion) + - name: effectiveLoggingVersion + value: $(loggingPackagePreviewVersion) + - name: effectiveAbstractionsVersion + value: $(abstractionsPackagePreviewVersion) + - name: effectiveSqlClientVersion + value: $(mdsPackagePreviewVersion) + - name: effectiveAzureVersion + value: $(azurePackagePreviewVersion) + - name: effectiveAkvProviderVersion + value: $(akvPackagePreviewVersion) - ${{ else }}: - - name: effectiveSqlServerVersion - value: $(sqlServerPackageVersion) - - name: effectiveLoggingVersion - value: $(loggingPackageVersion) - - name: effectiveAbstractionsVersion - value: $(abstractionsPackageVersion) - - name: effectiveSqlClientVersion - value: $(mdsPackageVersion) - - name: effectiveAzureVersion - value: $(azurePackageVersion) - - name: effectiveAkvProviderVersion - value: $(akvPackageVersion) + - name: effectiveSqlServerVersion + value: $(sqlServerPackageVersion) + - name: effectiveLoggingVersion + value: $(loggingPackageVersion) + - name: effectiveAbstractionsVersion + value: $(abstractionsPackageVersion) + - name: effectiveSqlClientVersion + value: $(mdsPackageVersion) + - name: effectiveAzureVersion + value: $(azurePackageVersion) + - name: effectiveAkvProviderVersion + value: $(akvPackageVersion) resources: repositories: @@ -132,8 +132,8 @@ extends: template: /v2/OneBranch.NonOfficial.CrossPlat.yml@templates parameters: release: - # This indicates the pipeline category to deploy Box products. See: - # https://eng.ms/docs/products/onebranch/release/yamlreleasepipelines/deployboxproducts + # This indicates the pipeline category to deploy Box products. See: + # https://eng.ms/docs/products/onebranch/release/yamlreleasepipelines/deployboxproducts category: NonAzure featureFlags: EnableCDPxPAT: false @@ -183,6 +183,7 @@ extends: parameters: debug: ${{ parameters.debug }} isPreview: ${{ parameters.isPreview }} + isOfficial: false # This is a non-official pipeline. publishSymbols: ${{ parameters.publishSymbols }} buildSqlServerServer: ${{ parameters.buildSqlServerServer }} buildSqlClient: ${{ parameters.buildSqlClient }} diff --git a/eng/pipelines/onebranch/sqlclient-official.yml b/eng/pipelines/onebranch/sqlclient-official.yml index c9548b25d6..1754918116 100644 --- a/eng/pipelines/onebranch/sqlclient-official.yml +++ b/eng/pipelines/onebranch/sqlclient-official.yml @@ -115,31 +115,31 @@ variables: # Define the effective versions for all of the packages we build and release. - ${{ if parameters.isPreview }}: - - name: effectiveSqlServerVersion - value: $(sqlServerPackagePreviewVersion) - - name: effectiveLoggingVersion - value: $(loggingPackagePreviewVersion) - - name: effectiveAbstractionsVersion - value: $(abstractionsPackagePreviewVersion) - - name: effectiveSqlClientVersion - value: $(mdsPackagePreviewVersion) - - name: effectiveAzureVersion - value: $(azurePackagePreviewVersion) - - name: effectiveAkvProviderVersion - value: $(akvPackagePreviewVersion) + - name: effectiveSqlServerVersion + value: $(sqlServerPackagePreviewVersion) + - name: effectiveLoggingVersion + value: $(loggingPackagePreviewVersion) + - name: effectiveAbstractionsVersion + value: $(abstractionsPackagePreviewVersion) + - name: effectiveSqlClientVersion + value: $(mdsPackagePreviewVersion) + - name: effectiveAzureVersion + value: $(azurePackagePreviewVersion) + - name: effectiveAkvProviderVersion + value: $(akvPackagePreviewVersion) - ${{ else }}: - - name: effectiveSqlServerVersion - value: $(sqlServerPackageVersion) - - name: effectiveLoggingVersion - value: $(loggingPackageVersion) - - name: effectiveAbstractionsVersion - value: $(abstractionsPackageVersion) - - name: effectiveSqlClientVersion - value: $(mdsPackageVersion) - - name: effectiveAzureVersion - value: $(azurePackageVersion) - - name: effectiveAkvProviderVersion - value: $(akvPackageVersion) + - name: effectiveSqlServerVersion + value: $(sqlServerPackageVersion) + - name: effectiveLoggingVersion + value: $(loggingPackageVersion) + - name: effectiveAbstractionsVersion + value: $(abstractionsPackageVersion) + - name: effectiveSqlClientVersion + value: $(mdsPackageVersion) + - name: effectiveAzureVersion + value: $(azurePackageVersion) + - name: effectiveAkvProviderVersion + value: $(akvPackageVersion) resources: repositories: @@ -153,8 +153,8 @@ extends: template: /v2/OneBranch.Official.CrossPlat.yml@templates parameters: release: - # This indicates the pipeline category to deploy Box products. See: - # https://eng.ms/docs/products/onebranch/release/yamlreleasepipelines/deployboxproducts + # This indicates the pipeline category to deploy Box products. See: + # https://eng.ms/docs/products/onebranch/release/yamlreleasepipelines/deployboxproducts category: NonAzure featureFlags: EnableCDPxPAT: false @@ -208,6 +208,7 @@ extends: parameters: debug: ${{ parameters.debug }} isPreview: ${{ parameters.isPreview }} + isOfficial: true # This is an official pipeline. publishSymbols: ${{ parameters.publishSymbols }} buildSqlServerServer: ${{ parameters.buildSqlServerServer }} buildSqlClient: ${{ parameters.buildSqlClient }} diff --git a/eng/pipelines/onebranch/stages/build-stages.yml b/eng/pipelines/onebranch/stages/build-stages.yml index 0b6962f1af..fad5c9a699 100644 --- a/eng/pipelines/onebranch/stages/build-stages.yml +++ b/eng/pipelines/onebranch/stages/build-stages.yml @@ -35,6 +35,11 @@ parameters: - name: isPreview type: boolean + # True if this is an official build, which runs additional ESRP malware scanning + # and codesigning steps. + - name: isOfficial + type: boolean + # True to publish symbols to public and private servers. - name: publishSymbols type: boolean @@ -60,40 +65,42 @@ stages: jobs: - ${{ if or(eq(parameters.buildAKVProvider, true), eq(parameters.buildSqlClient, true)) }}: - - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self - parameters: - packageName: Logging - packageFullName: Microsoft.Data.SqlClient.Internal.Logging - packageVersion: $(effectiveLoggingVersion) - versionProperties: >- - -p:LoggingPackageVersion=$(effectiveLoggingVersion) - -p:LoggingAssemblyFileVersion=$(loggingAssemblyFileVersion) - assemblyFileVersion: $(loggingAssemblyFileVersion) - publishSymbols: ${{ parameters.publishSymbols }} - esrpConnectedServiceName: $(ESRPConnectedServiceName) - esrpClientId: $(ESRPClientId) - appRegistrationClientId: $(AppRegistrationClientId) - appRegistrationTenantId: $(AppRegistrationTenantId) - authAkvName: $(AuthAKVName) - authSignCertName: $(AuthSignCertName) + - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self + parameters: + packageName: Logging + packageFullName: Microsoft.Data.SqlClient.Internal.Logging + packageVersion: $(effectiveLoggingVersion) + versionProperties: >- + -p:LoggingPackageVersion=$(effectiveLoggingVersion) + -p:LoggingAssemblyFileVersion=$(loggingAssemblyFileVersion) + assemblyFileVersion: $(loggingAssemblyFileVersion) + publishSymbols: ${{ parameters.publishSymbols }} + isOfficial: ${{ parameters.isOfficial }} + esrpConnectedServiceName: $(ESRPConnectedServiceName) + esrpClientId: $(ESRPClientId) + appRegistrationClientId: $(AppRegistrationClientId) + appRegistrationTenantId: $(AppRegistrationTenantId) + authAkvName: $(AuthAKVName) + authSignCertName: $(AuthSignCertName) - ${{ if eq(parameters.buildSqlServerServer, true) }}: - - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self - parameters: - packageName: SqlServer - packageFullName: Microsoft.SqlServer.Server - packageVersion: $(effectiveSqlServerVersion) - versionProperties: >- - -p:SqlServerAssemblyFileVersion=$(sqlServerAssemblyFileVersion) - -p:SqlServerPackageVersion=$(effectiveSqlServerVersion) - assemblyFileVersion: $(sqlServerAssemblyFileVersion) - publishSymbols: ${{ parameters.publishSymbols }} - esrpConnectedServiceName: $(ESRPConnectedServiceName) - esrpClientId: $(ESRPClientId) - appRegistrationClientId: $(AppRegistrationClientId) - appRegistrationTenantId: $(AppRegistrationTenantId) - authAkvName: $(AuthAKVName) - authSignCertName: $(AuthSignCertName) + - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self + parameters: + packageName: SqlServer + packageFullName: Microsoft.SqlServer.Server + packageVersion: $(effectiveSqlServerVersion) + versionProperties: >- + -p:SqlServerAssemblyFileVersion=$(sqlServerAssemblyFileVersion) + -p:SqlServerPackageVersion=$(effectiveSqlServerVersion) + assemblyFileVersion: $(sqlServerAssemblyFileVersion) + publishSymbols: ${{ parameters.publishSymbols }} + isOfficial: ${{ parameters.isOfficial }} + esrpConnectedServiceName: $(ESRPConnectedServiceName) + esrpClientId: $(ESRPClientId) + appRegistrationClientId: $(AppRegistrationClientId) + appRegistrationTenantId: $(AppRegistrationTenantId) + authAkvName: $(AuthAKVName) + authSignCertName: $(AuthSignCertName) # ==================================================================== # Stage 2: Abstractions package (depends on Logging from Stage 1) @@ -101,31 +108,32 @@ stages: # dependency on Internal.Logging. # ==================================================================== - ${{ if eq(parameters.buildSqlClient, true) }}: - - stage: build_abstractions - displayName: "Build Abstractions Package" - dependsOn: build_independent - - jobs: - - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self - parameters: - packageName: Abstractions - packageFullName: Microsoft.Data.SqlClient.Extensions.Abstractions - packageVersion: $(effectiveAbstractionsVersion) - versionProperties: >- - -p:AbstractionsPackageVersion=$(effectiveAbstractionsVersion) - -p:AbstractionsAssemblyFileVersion=$(abstractionsAssemblyFileVersion) - -p:LoggingPackageVersion=$(effectiveLoggingVersion) - assemblyFileVersion: $(abstractionsAssemblyFileVersion) - publishSymbols: ${{ parameters.publishSymbols }} - esrpConnectedServiceName: $(ESRPConnectedServiceName) - esrpClientId: $(ESRPClientId) - appRegistrationClientId: $(AppRegistrationClientId) - appRegistrationTenantId: $(AppRegistrationTenantId) - authAkvName: $(AuthAKVName) - authSignCertName: $(AuthSignCertName) - downloadArtifacts: - - artifactName: $(loggingArtifactsName) - displayName: Logging Package + - stage: build_abstractions + displayName: "Build Abstractions Package" + dependsOn: build_independent + + jobs: + - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self + parameters: + packageName: Abstractions + packageFullName: Microsoft.Data.SqlClient.Extensions.Abstractions + packageVersion: $(effectiveAbstractionsVersion) + versionProperties: >- + -p:AbstractionsPackageVersion=$(effectiveAbstractionsVersion) + -p:AbstractionsAssemblyFileVersion=$(abstractionsAssemblyFileVersion) + -p:LoggingPackageVersion=$(effectiveLoggingVersion) + assemblyFileVersion: $(abstractionsAssemblyFileVersion) + publishSymbols: ${{ parameters.publishSymbols }} + isOfficial: ${{ parameters.isOfficial }} + esrpConnectedServiceName: $(ESRPConnectedServiceName) + esrpClientId: $(ESRPClientId) + appRegistrationClientId: $(AppRegistrationClientId) + appRegistrationTenantId: $(AppRegistrationTenantId) + authAkvName: $(AuthAKVName) + authSignCertName: $(AuthSignCertName) + downloadArtifacts: + - artifactName: $(loggingArtifactsName) + displayName: Logging Package # ==================================================================== # Stage 3: Core packages (depend on Abstractions) @@ -133,88 +141,92 @@ stages: # Stage name kept as 'build_dependent' for validate job compatibility. # ==================================================================== - ${{ if eq(parameters.buildSqlClient, true) }}: - - stage: build_dependent - displayName: "Build Core Packages" - dependsOn: build_abstractions - - jobs: - - template: /eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml@self - parameters: - publishSymbols: ${{ parameters.publishSymbols }} - isPreview: ${{ parameters.isPreview }} - # TODO: This job should use the effective versions for Abstractions, Logging, - # SqlServer, and SqlClient. - - - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self - parameters: - packageName: Azure - packageFullName: Microsoft.Data.SqlClient.Extensions.Azure - packageVersion: $(effectiveAzureVersion) - versionProperties: >- - -p:AzurePackageVersion=$(effectiveAzureVersion) - -p:AzureAssemblyFileVersion=$(azureAssemblyFileVersion) - -p:AbstractionsPackageVersion=$(effectiveAbstractionsVersion) - -p:LoggingPackageVersion=$(effectiveLoggingVersion) - assemblyFileVersion: $(azureAssemblyFileVersion) - publishSymbols: ${{ parameters.publishSymbols }} - esrpConnectedServiceName: $(ESRPConnectedServiceName) - esrpClientId: $(ESRPClientId) - appRegistrationClientId: $(AppRegistrationClientId) - appRegistrationTenantId: $(AppRegistrationTenantId) - authAkvName: $(AuthAKVName) - authSignCertName: $(AuthSignCertName) - downloadArtifacts: - - artifactName: $(abstractionsArtifactsName) - displayName: Abstractions Package - - artifactName: $(loggingArtifactsName) - displayName: Logging Package + - stage: build_dependent + displayName: "Build Core Packages" + dependsOn: build_abstractions + + jobs: + - template: /eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml@self + parameters: + publishSymbols: ${{ parameters.publishSymbols }} + isPreview: ${{ parameters.isPreview }} + isOfficial: ${{ parameters.isOfficial }} + # TODO: This job should use the effective versions for Abstractions, Logging, + # SqlServer, and SqlClient. + + - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self + parameters: + packageName: Azure + packageFullName: Microsoft.Data.SqlClient.Extensions.Azure + packageVersion: $(effectiveAzureVersion) + versionProperties: >- + -p:AzurePackageVersion=$(effectiveAzureVersion) + -p:AzureAssemblyFileVersion=$(azureAssemblyFileVersion) + -p:AbstractionsPackageVersion=$(effectiveAbstractionsVersion) + -p:LoggingPackageVersion=$(effectiveLoggingVersion) + assemblyFileVersion: $(azureAssemblyFileVersion) + publishSymbols: ${{ parameters.publishSymbols }} + isOfficial: ${{ parameters.isOfficial }} + esrpConnectedServiceName: $(ESRPConnectedServiceName) + esrpClientId: $(ESRPClientId) + appRegistrationClientId: $(AppRegistrationClientId) + appRegistrationTenantId: $(AppRegistrationTenantId) + authAkvName: $(AuthAKVName) + authSignCertName: $(AuthSignCertName) + downloadArtifacts: + - artifactName: $(abstractionsArtifactsName) + displayName: Abstractions Package + - artifactName: $(loggingArtifactsName) + displayName: Logging Package # ==================================================================== # Stage 4: Add-on packages (depend on core packages) # AKV Provider builds after MDS completes. # ==================================================================== - ${{ if and(eq(parameters.buildAKVProvider, true), eq(parameters.buildSqlClient, true)) }}: - - stage: build_addons - displayName: "Build Add-on Packages" - dependsOn: build_dependent - - jobs: - - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self - parameters: - packageName: AkvProvider - packageFullName: Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider - packageVersion: $(effectiveAkvProviderVersion) - versionProperties: >- - -p:AkvPackageVersion=$(effectiveAkvProviderVersion) - -p:AkvAssemblyFileVersion=$(akvAssemblyFileVersion) - -p:MdsPackageVersion=$(effectiveSqlClientVersion) - -p:LoggingPackageVersion=$(effectiveLoggingVersion) - -p:AbstractionsPackageVersion=$(effectiveAbstractionsVersion) - assemblyFileVersion: $(akvAssemblyFileVersion) - publishSymbols: ${{ parameters.publishSymbols }} - esrpConnectedServiceName: $(ESRPConnectedServiceName) - esrpClientId: $(ESRPClientId) - appRegistrationClientId: $(AppRegistrationClientId) - appRegistrationTenantId: $(AppRegistrationTenantId) - authAkvName: $(AuthAKVName) - authSignCertName: $(AuthSignCertName) - downloadArtifacts: - - artifactName: $(sqlClientArtifactsName) - displayName: SqlClient Package - - artifactName: $(abstractionsArtifactsName) - displayName: Abstractions Package - - artifactName: $(loggingArtifactsName) - displayName: Logging Package + - stage: build_addons + displayName: "Build Add-on Packages" + dependsOn: build_dependent + + jobs: + - template: /eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml@self + parameters: + packageName: AkvProvider + packageFullName: Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider + packageVersion: $(effectiveAkvProviderVersion) + versionProperties: >- + -p:AkvPackageVersion=$(effectiveAkvProviderVersion) + -p:AkvAssemblyFileVersion=$(akvAssemblyFileVersion) + -p:MdsPackageVersion=$(effectiveSqlClientVersion) + -p:LoggingPackageVersion=$(effectiveLoggingVersion) + -p:AbstractionsPackageVersion=$(effectiveAbstractionsVersion) + assemblyFileVersion: $(akvAssemblyFileVersion) + publishSymbols: ${{ parameters.publishSymbols }} + isOfficial: ${{ parameters.isOfficial }} + esrpConnectedServiceName: $(ESRPConnectedServiceName) + esrpClientId: $(ESRPClientId) + appRegistrationClientId: $(AppRegistrationClientId) + appRegistrationTenantId: $(AppRegistrationTenantId) + authAkvName: $(AuthAKVName) + authSignCertName: $(AuthSignCertName) + downloadArtifacts: + - artifactName: $(sqlClientArtifactsName) + displayName: SqlClient Package + - artifactName: $(abstractionsArtifactsName) + displayName: Abstractions Package + - artifactName: $(loggingArtifactsName) + displayName: Logging Package # ==================================================================== # Validation # ==================================================================== - ${{ if eq(parameters.buildSqlClient, true) }}: - - stage: mds_package_validation - displayName: "MDS Package Validation" - dependsOn: build_dependent - jobs: - - template: /eng/pipelines/onebranch/jobs/validate-signed-package-job.yml@self - parameters: - artifactName: $(sqlClientArtifactsName) - isPreview: ${{ parameters.isPreview }} + - stage: mds_package_validation + displayName: "MDS Package Validation" + dependsOn: build_dependent + jobs: + - template: /eng/pipelines/onebranch/jobs/validate-signed-package-job.yml@self + parameters: + artifactName: $(sqlClientArtifactsName) + isPreview: ${{ parameters.isPreview }} + isOfficial: ${{ parameters.isOfficial }} diff --git a/eng/pipelines/onebranch/steps/compound-esrp-nuget-signing-step.yml b/eng/pipelines/onebranch/steps/compound-esrp-nuget-signing-step.yml index 34e903465f..824b3fab65 100644 --- a/eng/pipelines/onebranch/steps/compound-esrp-nuget-signing-step.yml +++ b/eng/pipelines/onebranch/steps/compound-esrp-nuget-signing-step.yml @@ -30,34 +30,39 @@ parameters: - name: esrpClientId type: string + # Glob pattern to match NuGet packages for scanning and signing. + - name: pattern + type: string + default: "*.*nupkg" + steps: # See: https://aka.ms/esrp.scantask - task: EsrpMalwareScanning@6 - displayName: 'ESRP Nuget Malware Scanning' + displayName: "ESRP Nuget Malware Scanning" inputs: - AppRegistrationClientId: '${{ parameters.appRegistrationClientId }}' - AppRegistrationTenantId: '${{ parameters.appRegistrationTenantId }}' + AppRegistrationClientId: "${{ parameters.appRegistrationClientId }}" + AppRegistrationTenantId: "${{ parameters.appRegistrationTenantId }}" CleanupTempStorage: 1 - ConnectedServiceName: '${{ parameters.esrpConnectedServiceName }}' - EsrpClientId: '${{ parameters.esrpClientId }}' - FolderPath: '$(PACK_OUTPUT)' - Pattern: '*.*nupkg' + ConnectedServiceName: "${{ parameters.esrpConnectedServiceName }}" + EsrpClientId: "${{ parameters.esrpClientId }}" + FolderPath: "$(PACK_OUTPUT)" + Pattern: "${{ parameters.pattern }}" UseMSIAuthentication: true VerboseLogin: 1 # See: https://aka.ms/esrp.signtask - task: EsrpCodeSigning@6 - displayName: 'ESRP Signing NuGet Package' + displayName: "ESRP Signing NuGet Package" inputs: - AppRegistrationClientId: '${{ parameters.appRegistrationClientId }}' - AppRegistrationTenantId: '${{ parameters.appRegistrationTenantId }}' - ConnectedServiceName: '${{ parameters.esrpConnectedServiceName }}' - EsrpClientId: '${{ parameters.esrpClientId }}' - AuthAKVName: '${{ parameters.authAkvName }}' - AuthSignCertName: '${{ parameters.authSignCertName }}' - FolderPath: '$(PACK_OUTPUT)' - Pattern: '*.*nupkg' - signConfigType: 'inlineSignParams' + AppRegistrationClientId: "${{ parameters.appRegistrationClientId }}" + AppRegistrationTenantId: "${{ parameters.appRegistrationTenantId }}" + ConnectedServiceName: "${{ parameters.esrpConnectedServiceName }}" + EsrpClientId: "${{ parameters.esrpClientId }}" + AuthAKVName: "${{ parameters.authAkvName }}" + AuthSignCertName: "${{ parameters.authSignCertName }}" + FolderPath: "$(PACK_OUTPUT)" + Pattern: "${{ parameters.pattern }}" + signConfigType: "inlineSignParams" UseMSIAuthentication: true inlineOperation: | [ diff --git a/eng/pipelines/onebranch/steps/esrp-code-signing-step.yml b/eng/pipelines/onebranch/steps/esrp-code-signing-step.yml index 59322d67aa..09f7715145 100644 --- a/eng/pipelines/onebranch/steps/esrp-code-signing-step.yml +++ b/eng/pipelines/onebranch/steps/esrp-code-signing-step.yml @@ -3,11 +3,12 @@ # The .NET Foundation licenses this file to you under the MIT license. # # See the LICENSE file in the project root for more information. # ################################################################################# + parameters: - name: artifactType values: - - dll - - pkg + - dll + - pkg - name: sourceRoot type: string @@ -15,7 +16,11 @@ parameters: - name: dllPattern type: string - default: 'Microsoft.Data.SqlClient*.dll' + default: "Microsoft.Data.SqlClient*.dll" + + - name: nupkgPattern + type: string + default: "*.*nupkg" - name: artifactDirectory type: string @@ -46,118 +51,118 @@ parameters: default: $(EsrpClientId) steps: -- ${{ if eq(parameters.artifactType, 'dll') }}: - # See: https://aka.ms/esrp.scantask - - task: EsrpMalwareScanning@6 - displayName: 'ESRP MalwareScanning' - inputs: - ConnectedServiceName: '${{parameters.ESRPConnectedServiceName }}' - AppRegistrationClientId: '${{parameters.appRegistrationClientId }}' - AppRegistrationTenantId: '${{parameters.appRegistrationTenantId }}' - EsrpClientId: '${{parameters.EsrpClientId }}' - UseMSIAuthentication: true - FolderPath: '${{parameters.sourceRoot }}' - Pattern: '${{ parameters.dllPattern }}' - CleanupTempStorage: 1 - VerboseLogin: 1 + # ESRP scan and sign the DLLs or NuGet packages, depending on the artifact type. + - ${{ if eq(parameters.artifactType, 'dll') }}: + # See: https://aka.ms/esrp.scantask + - task: EsrpMalwareScanning@6 + displayName: "ESRP MalwareScanning" + inputs: + ConnectedServiceName: "${{parameters.ESRPConnectedServiceName }}" + AppRegistrationClientId: "${{parameters.appRegistrationClientId }}" + AppRegistrationTenantId: "${{parameters.appRegistrationTenantId }}" + EsrpClientId: "${{parameters.EsrpClientId }}" + UseMSIAuthentication: true + FolderPath: "${{parameters.sourceRoot }}" + Pattern: "${{ parameters.dllPattern }}" + CleanupTempStorage: 1 + VerboseLogin: 1 - # See: https://aka.ms/esrp.signtask - - task: EsrpCodeSigning@6 - displayName: 'ESRP CodeSigning' - inputs: - ConnectedServiceName: '${{parameters.ESRPConnectedServiceName }}' - AppRegistrationClientId: '${{parameters.appRegistrationClientId }}' - AppRegistrationTenantId: '${{parameters.appRegistrationTenantId }}' - EsrpClientId: '${{parameters.EsrpClientId }}' - UseMSIAuthentication: true - AuthAKVName: '${{parameters.AuthAKVName }}' - AuthSignCertName: '${{parameters.AuthSignCertName }}' - FolderPath: '${{parameters.sourceRoot }}' - Pattern: '${{ parameters.dllPattern }}' - signConfigType: inlineSignParams - inlineOperation: | - [ - { - "keyCode": "CP-230012", - "operationSetCode": "SigntoolSign", - "parameters": [ - { - "parameterName": "OpusName", - "parameterValue": "Microsoft Data SqlClient Data Provider for SQL Server" - }, - { - "parameterName": "OpusInfo", - "parameterValue": "http://www.microsoft.com" - }, - { - "parameterName": "FileDigest", - "parameterValue": "/fd \"SHA256\"" - }, - { - "parameterName": "PageHash", - "parameterValue": "/NPH" - }, - { - "parameterName": "TimeStamp", - "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" - } - ], - "toolName": "sign", - "toolVersion": "1.0" - }, - { - "keyCode": "CP-230012", - "operationSetCode": "SigntoolVerify", - "parameters": [ ], - "toolName": "sign", - "toolVersion": "1.0" - } - ] + # See: https://aka.ms/esrp.signtask + - task: EsrpCodeSigning@6 + displayName: "ESRP CodeSigning" + inputs: + ConnectedServiceName: "${{parameters.ESRPConnectedServiceName }}" + AppRegistrationClientId: "${{parameters.appRegistrationClientId }}" + AppRegistrationTenantId: "${{parameters.appRegistrationTenantId }}" + EsrpClientId: "${{parameters.EsrpClientId }}" + UseMSIAuthentication: true + AuthAKVName: "${{parameters.AuthAKVName }}" + AuthSignCertName: "${{parameters.AuthSignCertName }}" + FolderPath: "${{parameters.sourceRoot }}" + Pattern: "${{ parameters.dllPattern }}" + signConfigType: inlineSignParams + inlineOperation: | + [ + { + "keyCode": "CP-230012", + "operationSetCode": "SigntoolSign", + "parameters": [ + { + "parameterName": "OpusName", + "parameterValue": "Microsoft Data SqlClient Data Provider for SQL Server" + }, + { + "parameterName": "OpusInfo", + "parameterValue": "http://www.microsoft.com" + }, + { + "parameterName": "FileDigest", + "parameterValue": "/fd \"SHA256\"" + }, + { + "parameterName": "PageHash", + "parameterValue": "/NPH" + }, + { + "parameterName": "TimeStamp", + "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + } + ], + "toolName": "sign", + "toolVersion": "1.0" + }, + { + "keyCode": "CP-230012", + "operationSetCode": "SigntoolVerify", + "parameters": [ ], + "toolName": "sign", + "toolVersion": "1.0" + } + ] -- ${{ if eq(parameters.artifactType, 'pkg') }}: - # See: https://aka.ms/esrp.scantask - - task: EsrpMalwareScanning@6 - displayName: 'ESRP MalwareScanning Nuget Package' - inputs: - ConnectedServiceName: '${{parameters.ESRPConnectedServiceName }}' - AppRegistrationClientId: '${{parameters.appRegistrationClientId }}' - AppRegistrationTenantId: '${{parameters.appRegistrationTenantId }}' - EsrpClientId: '${{parameters.EsrpClientId }}' - UseMSIAuthentication: true - FolderPath: '${{parameters.artifactDirectory }}' - Pattern: '*.*nupkg' - CleanupTempStorage: 1 - VerboseLogin: 1 + - ${{ if eq(parameters.artifactType, 'pkg') }}: + # See: https://aka.ms/esrp.scantask + - task: EsrpMalwareScanning@6 + displayName: "ESRP MalwareScanning Nuget Package" + inputs: + ConnectedServiceName: "${{parameters.ESRPConnectedServiceName }}" + AppRegistrationClientId: "${{parameters.appRegistrationClientId }}" + AppRegistrationTenantId: "${{parameters.appRegistrationTenantId }}" + EsrpClientId: "${{parameters.EsrpClientId }}" + UseMSIAuthentication: true + FolderPath: "${{parameters.artifactDirectory }}" + Pattern: "${{ parameters.nupkgPattern }}" + CleanupTempStorage: 1 + VerboseLogin: 1 - # See: https://aka.ms/esrp.signtask - - task: EsrpCodeSigning@6 - displayName: 'ESRP CodeSigning Nuget Package' - inputs: - inputs: - ConnectedServiceName: '${{parameters.ESRPConnectedServiceName }}' - AppRegistrationClientId: '${{parameters.appRegistrationClientId }}' - AppRegistrationTenantId: '${{parameters.appRegistrationTenantId }}' - EsrpClientId: '${{parameters.EsrpClientId }}' - UseMSIAuthentication: true - AuthAKVName: '${{parameters.AuthAKVName }}' - AuthSignCertName: '${{parameters.AuthSignCertName }}' - FolderPath: '${{parameters.artifactDirectory }}' - Pattern: '*.*nupkg' - signConfigType: inlineSignParams - inlineOperation: | - [ - { - "keyCode": "CP-401405", - "operationSetCode": "NuGetSign", - "parameters": [ ], - "toolName": "sign", - "toolVersion": "1.0" - }, - { - "keyCode": "CP-401405", - "operationSetCode": "NuGetVerify", - "parameters": [ ], - "toolName": "sign", - "toolVersion": "1.0" - } - ] + # See: https://aka.ms/esrp.signtask + - task: EsrpCodeSigning@6 + displayName: "ESRP CodeSigning Nuget Package" + inputs: + ConnectedServiceName: "${{parameters.ESRPConnectedServiceName }}" + AppRegistrationClientId: "${{parameters.appRegistrationClientId }}" + AppRegistrationTenantId: "${{parameters.appRegistrationTenantId }}" + EsrpClientId: "${{parameters.EsrpClientId }}" + UseMSIAuthentication: true + AuthAKVName: "${{parameters.AuthAKVName }}" + AuthSignCertName: "${{parameters.AuthSignCertName }}" + FolderPath: "${{parameters.artifactDirectory }}" + Pattern: "${{ parameters.nupkgPattern }}" + signConfigType: inlineSignParams + inlineOperation: | + [ + { + "keyCode": "CP-401405", + "operationSetCode": "NuGetSign", + "parameters": [ ], + "toolName": "sign", + "toolVersion": "1.0" + }, + { + "keyCode": "CP-401405", + "operationSetCode": "NuGetVerify", + "parameters": [ ], + "toolName": "sign", + "toolVersion": "1.0" + } + ]