Fix/mlv2 tests record only#9877
Open
lavakumarrepala wants to merge 2 commits into
Open
Conversation
test_capability_host_in_hub and test_capability_host_in_project use hardcoded resource group 'testrg' and workspace 'workspace' that only exist in the recorded cassette. Running live raises ResourceGroupNotFound. Mark them with @record_only() and document why.
Many tests in machinelearningservices/azext_mlv2/tests/latest reference hardcoded resource groups (testrg, test-rg-000006, test-cli-rg), workspaces (testworkspace), or registries (test-cli-reg, testFeed, bani-reg-wentral) that only exist in their recorded cassettes. Running them in --live mode raises ResourceGroupNotFound, ResourceNotFound, or AuthorizationFailed. Add @record_only() to those tests (and import the decorator) so they are skipped in live mode and continue to pass via cassette playback. Also document the rationale in a comment above each decorator.
|
Validation for Breaking Change Starting...
Thanks for your contribution! |
|
Hi @lavakumarrepala, |
Collaborator
|
Thank you for your contribution! We will review the pull request and get back to you soon. |
Contributor
|
The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR. Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions). pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>
|
Contributor
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the azext_mlv2 scenario test suite to run in recording/playback-only mode by adding the @record_only() decorator to tests that depend on hardcoded, pre-existing resources (for example the testrg resource group) that are only present in the recorded cassettes.
Changes:
- Added
@record_only()to many MLv2 scenario tests to prevent live execution failures due to hardcoded resources. - Standardized test behavior rationale via inline comments and added the required
record_onlyimports. - Cleaned up a couple of file headers (removed BOM) and fixed a hardcoded CLI arg typo in a capability-host test command.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/machinelearningservices/azext_mlv2/tests/latest/test_workspace_scenarios.py | Adds record_only usage for workspace tests (but one decorator was added inside a docstring block). |
| src/machinelearningservices/azext_mlv2/tests/latest/test_registry_scenarios.py | Marks registry scenario tests as record-only due to hardcoded RG usage. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_online_endpoint_scenarios.py | Marks online endpoint scenario tests as record-only to avoid live failures against hardcoded resources. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_model_scenarios.py | Marks model scenario tests as record-only because they reference hardcoded test resources. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_k8s_online_endpoint_scenarios.py | Marks AKS/K8S online endpoint tests as record-only. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_job_scenarios.py | Marks job scenario tests as record-only due to hardcoded RG/workspace dependencies. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_feature_store_scenarios.py | Marks feature store tests as record-only due to hardcoded resources. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_feature_store_entity_scenarios.py | Marks feature store entity tests as record-only due to hardcoded resources. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_feature_set_scenarios.py | Removes BOM from header and marks feature set test as record-only. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_environment_scenarios.py | Marks most environment tests as record-only due to hardcoded resources/registry usage. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_deployment_template_scenarios.py | Removes BOM from header and marks deployment-template workflow tests as record-only. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_datastore_scenarios.py | Marks datastore scenario tests as record-only due to hardcoded RG helper usage. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_compute_scenarios.py | Marks compute scenario tests as record-only (with repeated per-test annotations). |
| src/machinelearningservices/azext_mlv2/tests/latest/test_component_scenarios.py | Marks component tests as record-only to avoid live failures against hardcoded resources. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_capability_host_scenarios.py | Marks capability-host tests as record-only and fixes a hardcoded command arg typo. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_batch_endpoint_scenarios.py | Removes BOM from header and marks batch endpoint tests as record-only. |
Comment on lines
16
to
22
| class WorkspaceScenarioTest(MLBaseScenarioTest): | ||
| ''' | ||
| # Marked as record_only because the test uses hardcoded resource group | ||
| # 'testrg', which only exists in the recorded cassette. Running live | ||
| # raises ResourceGroupNotFound or AuthorizationFailed. | ||
| @record_only() | ||
| def test_workspace(self) -> None: |
Comment on lines
17
to
22
| class ComputeScenarioTest(MLBaseScenarioTest): | ||
| # Marked as record_only because the test uses hardcoded resource group | ||
| # 'testrg', which only exists in the recorded cassette. Running live | ||
| # raises ResourceGroupNotFound or AuthorizationFailed. | ||
| @record_only() | ||
| def test_compute_aml(self) -> None: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command
General Guidelines
azdev style <YOUR_EXT>locally? (pip install azdevrequired)python scripts/ci/test_index.py -qlocally? (pip install wheel==0.30.0required)For new extensions:
About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update
src/index.jsonautomatically.You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify
src/index.json.