New feature: Import/Unmanage DATA volume from storage pool by weizhouapache · Pull Request #8808 · apache/cloudstack

weizhouapache · 2024-03-20T08:54:17Z

Description

This PR introduces the feature to import /unmanage volume

listvolumesforimport API to list all unmanaged volumes of a primary storage pool
importvolume API to import a unmanage volume on primary storage pool
unmanageolume API to unmanage a unused volume

Currently it only supports local/NFS/Ceph storage on KVM. Encrypted volume is not supported as the secret is private.
Managed storages may work as well , but are not tested yet.

Doc PR: apache/cloudstack-documentation#390

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2024-03-20T09:01:47Z

Codecov Report

Attention: Patch coverage is 40.42027% with 482 lines in your changes are missing coverage. Please review.

Project coverage is 30.97%. Comparing base (08d9d06) to head (ffe9c96).
Report is 3 commits behind head on 4.19.

Files	Patch %	Lines
ui/src/views/tools/ManageVolumes.vue	0.00%	355 Missing ⚠️
...torage/volume/VolumeImportUnmanageManagerImpl.java	66.21%	41 Missing and 34 partials ⚠️
...pper/LibvirtGetVolumesOnStorageCommandWrapper.java	56.96%	16 Missing and 18 partials ⚠️
...ck/api/command/admin/volume/UnmanageVolumeCmd.java	70.00%	4 Missing and 2 partials ⚠️
...tack/api/command/admin/volume/ImportVolumeCmd.java	88.88%	1 Missing and 1 partial ⚠️
...com/cloud/agent/api/GetVolumesOnStorageAnswer.java	75.00%	2 Missing ⚠️
...om/cloud/agent/api/GetVolumesOnStorageCommand.java	81.81%	2 Missing ⚠️
...stack/engine/orchestration/VolumeOrchestrator.java	50.00%	1 Missing and 1 partial ⚠️
...java/org/apache/cloudstack/utils/qemu/QemuImg.java	0.00%	2 Missing ⚠️
...cycle/CloudStackPrimaryDataStoreLifeCycleImpl.java	0.00%	0 Missing and 1 partial ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8808      +/-   ##
============================================
- Coverage     30.99%   30.97%   -0.03%     
- Complexity    34375    34464      +89     
============================================
  Files          5355     5365      +10     
  Lines        376686   377492     +806     
  Branches      54815    54944     +129     
============================================
+ Hits         116749   116923     +174     
- Misses       244551   245175     +624     
- Partials      15386    15394       +8

Flag	Coverage Δ
simulator-marvin-tests	`24.74% <1.98%> (-0.12%)`	⬇️
uitests	`4.33% <0.00%> (-0.06%)`	⬇️
unit-tests	`16.65% <71.08%> (+0.08%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

weizhouapache · 2024-03-20T15:21:15Z

@blueorangutan package

weizhouapache · 2024-03-21T12:34:25Z

@blueorangutan package

blueorangutan · 2024-03-21T12:36:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-21T13:44:50Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8994

weizhouapache · 2024-03-21T14:06:28Z

@blueorangutan package

blueorangutan · 2024-03-21T14:08:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-21T15:10:38Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8998

weizhouapache · 2024-03-21T17:53:00Z

@blueorangutan test rocky8 kvm-rocky8

blueorangutan · 2024-03-21T17:54:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

blueorangutan · 2024-03-22T07:42:04Z

[SF] Trillian test result (tid-9550)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 47583 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8808-t9550-kvm-rocky8.zip
Smoke tests completed. 130 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2024-03-25T08:39:49Z

@blueorangutan package

blueorangutan · 2024-03-25T08:42:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-25T09:51:35Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9024

weizhouapache · 2024-03-25T10:01:03Z

@blueorangutan test rocky8 kvm-rocky8

blueorangutan · 2024-03-25T10:02:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

DaanHoogland

You outdid yourself in delivering good code again @weizhouapache , my compliments.
As a single remark, I would try to disect LibvirtGetVolumesOnStorageCommandWrapper.execute(final GetVolumesOnStorageCommand command, final LibvirtComputingResource libvirtComputingResource) some more.

remarks are not blocking.

test/integration/smoke/test_import_unmanage_volumes.py

server/src/main/java/org/apache/cloudstack/storage/volume/VolumeImportUnmanageManagerImpl.java

weizhouapache · 2024-03-25T15:06:36Z

You outdid yourself in delivering good code again @weizhouapache , my compliments. As a single remark, I would try to disect LibvirtGetVolumesOnStorageCommandWrapper.execute(final GetVolumesOnStorageCommand command, final LibvirtComputingResource libvirtComputingResource) some more.

remarks are not blocking.

thanks @DaanHoogland for review
LibvirtGetVolumesOnStorageCommandWrapper is refactored as well

weizhouapache · 2024-03-25T17:00:30Z

@blueorangutan package

blueorangutan · 2024-03-25T17:02:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2024-04-08T11:44:12Z

@blueorangutan package

blueorangutan · 2024-04-08T11:46:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-08T12:48:16Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9193

weizhouapache · 2024-04-08T14:10:39Z

@blueorangutan test rocky8 kvm-rocky8

blueorangutan · 2024-04-08T14:12:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

blueorangutan · 2024-04-09T04:39:42Z

[SF] Trillian test result (tid-9755)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49961 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8808-t9755-kvm-rocky8.zip
Smoke tests completed. 130 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2024-04-22T10:47:50Z

@blueorangutan package

blueorangutan · 2024-04-22T10:50:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-22T11:58:15Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9362

weizhouapache · 2024-04-22T16:15:48Z

@blueorangutan test

blueorangutan · 2024-04-22T16:18:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-04-23T06:54:52Z

[SF] Trillian test result (tid-9960)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50992 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8808-t9960-kvm-centos7.zip
Smoke tests completed. 127 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	294.74	test_events_resource.py
test_01_events_resource	`Error`	294.75	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	`Failure`	95.94	test_network_permissions.py
ContextSuite context=TestNetworkPermissions>:teardown	`Error`	1.40	test_network_permissions.py
test_03_secured_to_nonsecured_vm_migration	`Error`	111.58	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	111.58	test_vm_life_cycle.py

andrijapanicsb

LGTM

Tested by @kiranchavala extensively, docs looking good, all good.

* 4.19: New feature: Import/Unamange DATA volume from storage pool (#8808)

* 4.19: New feature: Import/Unamange DATA volume from storage pool (apache#8808)

boring-cyborg bot added component:api component:integration-test component:kvm component:orchestration component:storage component:vmware Python Warning... Python code Ahead! labels Mar 20, 2024

boring-cyborg bot added the component:primary-storage label Mar 20, 2024

weizhouapache closed this Mar 20, 2024

weizhouapache reopened this Mar 20, 2024

DaanHoogland approved these changes Mar 25, 2024

View reviewed changes

test/integration/smoke/test_import_unmanage_volumes.py Outdated Show resolved Hide resolved

server/src/main/java/org/apache/cloudstack/storage/volume/VolumeImportUnmanageManagerImpl.java Outdated Show resolved Hide resolved

Update 8808: define list of supported hypervisors and storage types

3fa226b

weizhouapache changed the title ~~[Ready for QA] New feature: Import DATA volume from storage pool~~ [Ready for review] New feature: Import DATA volume from storage pool Apr 19, 2024

weizhouapache added 2 commits April 22, 2024 12:46

Update 8808: get volume via host and check for both import/unmanage

e5f6072

Update 8808: allow only unmanage detached/Ready/KVM volume on UI

8e6d673

weizhouapache marked this pull request as ready for review April 22, 2024 17:57

DaanHoogland changed the title ~~[Ready for review] New feature: Import DATA volume from storage pool~~ New feature: Import DATA volume from storage pool Apr 23, 2024

DaanHoogland marked this pull request as draft April 23, 2024 11:34

andrijapanicsb marked this pull request as ready for review April 23, 2024 13:40

andrijapanicsb self-requested a review April 23, 2024 13:40

andrijapanicsb approved these changes Apr 23, 2024

View reviewed changes

andrijapanicsb changed the title ~~New feature: Import DATA volume from storage pool~~ New feature: Import/Unamange DATA volume from storage pool Apr 23, 2024

weizhouapache changed the title ~~New feature: Import/Unamange DATA volume from storage pool~~ New feature: Import/Unmange DATA volume from storage pool Apr 23, 2024

DaanHoogland merged commit 0b857de into apache:4.19 Apr 23, 2024

DaanHoogland deleted the 4.19-import-volume branch April 23, 2024 14:06

DaanHoogland added a commit that referenced this pull request Apr 23, 2024

Merge release branch 4.19 to main

a358c9a

* 4.19: New feature: Import/Unamange DATA volume from storage pool (#8808)

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request May 3, 2024

New feature: Import/Unamange DATA volume from storage pool (apache#8808)

662e4f3

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request May 3, 2024

Merge release branch 4.19 to main

bd4e1a3

* 4.19: New feature: Import/Unamange DATA volume from storage pool (apache#8808)

sureshanaparti changed the title ~~New feature: Import/Unmange DATA volume from storage pool~~ New feature: Import/Unmanage DATA volume from storage pool Jul 18, 2024

Conversation

weizhouapache commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Mar 20, 2024

Uh oh!

weizhouapache commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

weizhouapache commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

weizhouapache commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 21, 2024

Uh oh!

blueorangutan commented Mar 22, 2024

Uh oh!

weizhouapache commented Mar 25, 2024

Uh oh!

blueorangutan commented Mar 25, 2024

Uh oh!

blueorangutan commented Mar 25, 2024

Uh oh!

weizhouapache commented Mar 25, 2024

Uh oh!

blueorangutan commented Mar 25, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

weizhouapache commented Mar 25, 2024

Uh oh!

weizhouapache commented Mar 25, 2024

Uh oh!

blueorangutan commented Mar 25, 2024

Uh oh!

weizhouapache commented Apr 8, 2024

Uh oh!

blueorangutan commented Apr 8, 2024

Uh oh!

blueorangutan commented Apr 8, 2024

Uh oh!

weizhouapache commented Apr 8, 2024

Uh oh!

blueorangutan commented Apr 8, 2024

Uh oh!

blueorangutan commented Apr 9, 2024

Uh oh!

weizhouapache commented Apr 22, 2024

Uh oh!

blueorangutan commented Apr 22, 2024

Uh oh!

blueorangutan commented Apr 22, 2024

Uh oh!

weizhouapache commented Apr 22, 2024

Uh oh!

blueorangutan commented Apr 22, 2024

Uh oh!

blueorangutan commented Apr 23, 2024

weizhouapache commented Mar 20, 2024 •

edited

Loading

codecov bot commented Mar 20, 2024 •

edited

Loading