Add SharedMountPoint to KVMs supported storage pool types by GutoVeronezi · Pull Request #4780 · apache/cloudstack

GutoVeronezi · 2021-03-09T21:07:09Z

Description

ACS, with KVM, cannot migrate VM's volumes from a SharedMountPoint to a NFS storage pool properly; it will not copy the volumes's template to the destination storage pool (if needed), neither update the VM's XML properly.

From a SharedMountPoint to a NFS storage pool, ACS used to interpret that it would use localhost as the host of the storage system; as it is a shared mount point configured in all cluster's host, as server name, but NFS retrieves its path from KVM, so it needs to keep the same behavior as NFS to NFS.

This PR intends to enable migration between NFS <> ISCSI (via sharedmountpoint) in KVM.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

How Has This Been Tested?

It has been tested locally in a test lab.

I created a VM;
I configured 2 primary storages, 1 as NFS and 1 as SharedMountPoint;
I tried to migrate VM (with migrateVirtualMachineWithVolume) from a SharedMountPoint to a NFS, but it threw a Exception due to incorrect path on VM's XML;

-- After appling this patch the migration works.

DaanHoogland · 2021-03-11T08:31:37Z

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

+        List<StoragePoolType> values = new ArrayList<>();
+
+        values.add(StoragePoolType.NetworkFilesystem);
+        values.add(StoragePoolType.SharedMountPoint);


is this true for all hypervisors/storage backends? I know this is a new method only called from the KVMNonManaged, but that is not reflected in the name or protected by encapsulation. (genuine question)

Actually KVM is the only hypervisor that enables SharedMountPoint as an option, so it is true to all hypervisors that has it enabled.

DaanHoogland

clgtm, ...

GutoVeronezi · 2021-05-24T13:15:27Z

Can anyone review this?

GutoVeronezi · 2021-06-07T13:05:22Z

Can anyone review this?

GabrielBrascher

Code LGTM

blueorangutan · 2021-06-10T15:37:06Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 201

blueorangutan · 2021-06-12T03:06:38Z

Trillian test result (tid-920)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 59438 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t920-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 83 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Error`	538.65	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	280.41	test_internal_lb.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1041.13	test_privategw_acl.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3612.99	test_kubernetes_clusters.py
test_02_deploy_and_upgrade_kubernetes_cluster	`Failure`	3607.88	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_04_basic_lifecycle_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	98.09	test_kubernetes_clusters.py
test_02_isolate_network_FW_PF_default_routes_egress_false	`Failure`	302.18	test_routers_network_ops.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	525.17	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	526.33	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	473.81	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	473.82	test_vpc_redundant.py

yadvr · 2021-06-17T07:52:01Z

@blueorangutan package

blueorangutan · 2021-06-17T07:53:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-06-17T09:09:06Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 260

GutoVeronezi · 2021-06-21T13:54:57Z

@DaanHoogland @GabrielBrascher is there anything else to do?

DaanHoogland · 2021-06-22T06:44:15Z

@GutoVeronezi no, just a test run that shows no regressions. I just started it.

blueorangutan · 2021-06-22T07:40:21Z

Trillian Build Failed (tid-1034)

blueorangutan · 2021-06-23T22:39:26Z

Trillian test result (tid-1063)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34706 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1063-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Smoke tests completed. 88 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2021-06-24T05:41:20Z

Trillian test result (tid-1068)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50300 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1068-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 83 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	348.48	test_routers_network_ops.py
test_01_volume_usage	`Failure`	787.27	test_usage.py
test_10_attachAndDetach_iso	`Failure`	1510.22	test_vm_life_cycle.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	539.08	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	514.84	test_vpc_redundant.py
test_01_vpc_site2site_vpn_multiple_options	`Failure`	670.27	test_vpc_vpn.py

GutoVeronezi · 2021-06-28T16:26:19Z

@DaanHoogland any update about these tests results?

DaanHoogland · 2021-06-29T07:21:42Z

@DaanHoogland any update about these tests results?

@GutoVeronezi sorry no, too much testing going on and it isn't even my main focus.

GutoVeronezi · 2021-07-20T12:48:19Z

Any update about this?

blueorangutan · 2021-07-20T14:24:08Z

Packaging result: ✖️ el7 ✔️ el8 ✔️ debian. SL-JID 602

blueorangutan · 2021-07-21T08:49:07Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 615

blueorangutan · 2021-07-22T22:02:06Z

Trillian test result (tid-1350)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49986 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1350-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 86 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_router_dhcphosts	`Failure`	11.58	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPHosts>:teardown	`Error`	20.85	test_router_dhcphosts.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	346.23	test_routers_network_ops.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	541.98	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	475.82	test_vpc_redundant.py

blueorangutan · 2021-07-23T17:09:53Z

Trillian test result (tid-1359)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33472 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1359-kvm-centos7.zip
Smoke tests completed. 89 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

GutoVeronezi · 2021-07-27T22:59:19Z

@blueorangutan package

blueorangutan · 2021-07-27T23:00:03Z

@GutoVeronezi a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-07-27T23:25:42Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 662

blueorangutan · 2021-07-29T03:14:40Z

Trillian test result (tid-1394)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47859 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1394-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 87 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	380.22	test_privategw_acl.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	560.78	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	548.82	test_vpc_redundant.py

DaanHoogland · 2021-07-29T12:15:17Z

these errors seem consistent @GutoVeronezi . can you have a look?

GutoVeronezi · 2021-08-09T18:55:04Z

@DaanHoogland could we run the tests one more time?

GutoVeronezi · 2021-08-09T18:55:18Z

@blueorangutan package

blueorangutan · 2021-08-09T18:56:02Z

@GutoVeronezi a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-08-09T19:41:27Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 816

DaanHoogland · 2021-08-11T08:24:53Z

@blueorangutan test

blueorangutan · 2021-08-11T08:26:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-08-12T01:14:39Z

Trillian test result (tid-1590)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 58538 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1590-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Smoke tests completed. 85 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_07_deploy_kubernetes_ha_cluster	`Failure`	3627.77	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.07	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	44.79	test_kubernetes_clusters.py
test_04_nonsecured_to_secured_vm_migration	`Error`	467.06	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	0.04	test_vm_life_cycle.py
test_01_create_volume	`Failure`	782.26	test_volumes.py
test_02_VPC_default_routes	`Failure`	237.52	test_vpc_router_nics.py

nvazquez · 2021-08-16T02:58:46Z

@blueorangutan test

blueorangutan · 2021-08-16T02:59:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-08-16T14:53:56Z

Trillian test result (tid-1646)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40976 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4780-t1646-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 89 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez

LGTM

Add SharedMountPoint to KVMs supported storage pool types

5bf1d73

DaanHoogland reviewed Mar 11, 2021

View reviewed changes

DaanHoogland approved these changes Mar 12, 2021

View reviewed changes

GabrielBrascher approved these changes Jun 8, 2021

View reviewed changes

GutoVeronezi marked this pull request as draft June 11, 2021 21:10

Fix live migration to iSCSI and improve logs

2f6b9cb

GutoVeronezi marked this pull request as ready for review June 15, 2021 18:10

GabrielBrascher added this to the 4.16.0.0 milestone Jun 29, 2021

GabrielBrascher added component:primary-storage component:kvm labels Jun 29, 2021

GabrielBrascher assigned GutoVeronezi Jun 29, 2021

nvazquez added the status:needs-testing label Jul 13, 2021

nvazquez approved these changes Aug 16, 2021

View reviewed changes

nvazquez merged commit 65a48dc into apache:main Aug 16, 2021

DaanHoogland removed the status:needs-testing label Jan 28, 2026

Conversation

GutoVeronezi commented Mar 9, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

How Has This Been Tested?

Uh oh!

DaanHoogland Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi commented May 24, 2021

Uh oh!

GutoVeronezi commented Jun 7, 2021

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 10, 2021

Uh oh!

blueorangutan commented Jun 12, 2021

Uh oh!

yadvr commented Jun 17, 2021

Uh oh!

blueorangutan commented Jun 17, 2021

Uh oh!

blueorangutan commented Jun 17, 2021

Uh oh!

GutoVeronezi commented Jun 21, 2021

Uh oh!

DaanHoogland commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 23, 2021

Uh oh!

blueorangutan commented Jun 24, 2021

Uh oh!

GutoVeronezi commented Jun 28, 2021

Uh oh!

DaanHoogland commented Jun 29, 2021

Uh oh!

GutoVeronezi commented Jul 20, 2021

Uh oh!

blueorangutan commented Jul 20, 2021

Uh oh!

blueorangutan commented Jul 21, 2021

Uh oh!

blueorangutan commented Jul 22, 2021

Uh oh!

blueorangutan commented Jul 23, 2021

Uh oh!

GutoVeronezi commented Jul 27, 2021

Uh oh!

blueorangutan commented Jul 27, 2021

Uh oh!

blueorangutan commented Jul 27, 2021

Uh oh!

blueorangutan commented Jul 29, 2021

Uh oh!

DaanHoogland commented Jul 29, 2021

Uh oh!

GutoVeronezi commented Aug 9, 2021

Uh oh!

GutoVeronezi commented Aug 9, 2021

Uh oh!

blueorangutan commented Aug 9, 2021

Uh oh!

blueorangutan commented Aug 9, 2021

Uh oh!

DaanHoogland commented Aug 11, 2021

Uh oh!

blueorangutan commented Aug 11, 2021

Uh oh!