Cannot live migrate XenServer VMs between two storage pools

[dpassante]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Hypervisor Controler

CLOUDSTACK VERSION

4.11

CONFIGURATION

XenServer clusters with local storage or DefaultPrimary PreSetup SRs.

OS / ENVIRONMENT

N/A

SUMMARY

Trying to migrate a XenServer VM to another storage pool causes a "Can not see storage pool" exception.

This occurs because the SR's path is used to query the SRs by name label on the XenServer host.

STEPS TO REPRODUCE

cs --region preprod migrateVirtualMachineWithVolume virtualmachineid=e731afa7-7aab-4c4b-ab73-0e756dab8cfb hostid=529e6778-5e19-4ff0-806d-680451f5d4d7

EXPECTED RESULTS

Migration success.

ACTUAL RESULTS

2018-06-01 16:26:30,222 ERROR [c.c.h.x.r.w.x.XenServer610MigrateWithStorageReceiveCommandWrapper] (DirectAgent-4:ctx-1e92a910) (logid:bdc18e3b) Migration of
vm i-2-1946-VM with storage failed due to com.cloud.utils.exception.CloudRuntimeException: Can not see storage pool: ext from on host:db5ea702-5d01-4bcd-b736
-25c80a506a5e
com.cloud.utils.exception.CloudRuntimeException: Can not see storage pool: ext from on host:db5ea702-5d01-4bcd-b736-25c80a506a5e
        at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageReceiveCommandWrapper.execute(XenServer610MigrateWithStorageR
eceiveCommandWrapper.java:73)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageReceiveCommandWrapper.execute(XenServer610MigrateWithStorageR
eceiveCommandWrapper.java:48)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-06-01 16:26:30,223 ERROR [o.a.c.s.m.XenServerStorageMotionStrategy] (Work-Job-Executor-2:ctx-3df539ea job-6147/job-6148 ctx-00f4325c) (logid:bdc18e3b) M
igration with storage of vm VM[User|i-2-1946-VM] failed. Details: Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Can not see storage pool: ext from on host:db5ea702-5d01-4bcd-b736-25c80a506a5e
Stack: com.cloud.utils.exception.CloudRuntimeException: Can not see storage pool: ext from on host:db5ea702-5d01-4bcd-b736-25c80a506a5e
        at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageReceiveCommandWrapper.execute(XenServer610MigrateWithStorageR
eceiveCommandWrapper.java:73)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageReceiveCommandWrapper.execute(XenServer610MigrateWithStorageR
eceiveCommandWrapper.java:48)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

2018-06-01 16:26:30,223 ERROR [o.a.c.s.m.XenServerStorageMotionStrategy] (Work-Job-Executor-2:ctx-3df539ea job-6147/job-6148 ctx-00f4325c) (logid:bdc18e3b) c
opy failed
com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|i-2-1946-VM] to host Host[-23-Routing]
        at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(XenServerStorageMotionStrategy.java:346)
        at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:122)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1646)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1054)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2553)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4987)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5090)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[rafaelweingartner]

Do you have other data disks attached in this VM?

[dpassante]

@rafaelweingartner Yes, the VM have data disk attached but I'm having the same issue without datadisk.

[rafaelweingartner]

I have fixed this problem in master (even with datadisks), but I am not sure if this was backported. I will check the code in 4.11.1.0.

[dpassante]

Great news! It's a bit of a bloquer issue for us and I'd like to see it fixed on the next release :-)

[DaanHoogland]

do you have a ref to the PR to add here, @rafaelweingartner ?

[rafaelweingartner]

@DaanHoogland not yet. I am traveling today. Probably only in the weekend I would be able to take a look into it as it was not going to be a Day job thing. I have fixed this issue in master (even for VMs with data disk attached) though.

[DaanHoogland]

@rafaelweingartner @dpassante I am marking this for 4.11.2 as it is not planned and not a regression since 4.11.0

[yadvr]

I've validated the issue, the fix LGTM: #2686