ISCSI cleanup thread breaks non-managed storage utilizing ISCSI connectivity #4210
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
KVM plugin
CLOUDSTACK VERSION
4.13.1 and 4.14
CONFIGURATION
KVM with unmanaged ISCSI
OS / ENVIRONMENT
SUMMARY
Change was introduced in 6baa598
It adds a cleanup process for old iscsi connections. Unfortunately, it doesn't appear to be scoped and makes the assumption that all iscsi is managed. When working with a community member on slack who upgraded to 4.13.1 found that it broke their unmanaged ISCSI storage.
STEPS TO REPRODUCE
Upgrade an unmanaged ISCSI based system to 4.13.1 and your iscsi will be ripped out when the agent is started.
ACTUAL RESULTS
2020-07-14 21:50:21,607 DEBUG [kvm.resource.LibvirtConnection] (Thread-6:null) (logid:) Looking for libvirtd connection at: qemu:///system
2020-07-14 21:50:21,610 DEBUG [kvm.storage.IscsiStorageCleanupMonitor] (Thread-6:null) (logid:) found iscsi disk by cleanup thread, marking inactive: /dev/disk/by-path/ip-192.168.10.10:3260-iscsi-iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1-lun-0
2020-07-14 21:50:21,611 DEBUG [kvm.storage.IscsiStorageCleanupMonitor] (Thread-6:null) (logid:) found 0 domains
2020-07-14 21:50:21,617 INFO [kvm.storage.IscsiStorageCleanupMonitor] (Thread-6:null) (logid:) Cleaning up disk /dev/disk/by-path/ip-192.168.10.10:3260-iscsi-iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1-lun-0
2020-07-14 21:50:21,618 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Executing: sudo iscsiadm -m node -T iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1 -p 192.168.10.10:3260 --logout
2020-07-14 21:50:21,623 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Executing while with timeout : 3600000
2020-07-14 21:50:21,658 INFO [cloud.agent.Agent] (main:null) (logid:) Connecting to host:192.168.1.108
2020-07-14 21:50:21,659 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.1.108:8250
2020-07-14 21:50:21,659 WARN [utils.nio.NioConnection] (main:null) (logid:) Unable to connect to remote: is there a server running on port 8250
2020-07-14 21:50:21,759 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Execution is successful.
2020-07-14 21:50:21,760 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Logging out of session [sid: 1, target: iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1, portal: 192.168.10.10,3260]
Logging out of session [sid: 2, target: iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1, portal: 192.168.10.10,3260]
Logout of [sid: 1, target: iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1, portal: 192.168.10.10,3260] successful.
Logout of [sid: 2, target: iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1, portal: 192.168.10.10,3260] successful.
2020-07-14 21:50:21,761 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Successfully logged out of iSCSI target /iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1/0
2020-07-14 21:50:21,761 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Executing: sudo iscsiadm -m node -T iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1 -p 192.168.10.10:3260 -o delete
2020-07-14 21:50:21,762 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Executing while with timeout : 3600000
2020-07-14 21:50:21,787 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Execution is successful.
2020-07-14 21:50:21,788 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Removed iSCSI target /iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1/0
2020-07-14 21:50:21,789 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Executing: sudo blockdev --getsize64 /dev/disk/by-path/ip-192.168.10.10:3260-iscsi-iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1-lun-0
2020-07-14 21:50:21,790 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Executing while with timeout : 3600000
2020-07-14 21:50:21,813 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Exit value is 1
2020-07-14 21:50:21,814 DEBUG [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) blockdev: cannot open /dev/disk/by-path/ip-192.168.10.10:3260-iscsi-iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1-lun-0: No such file or directory
2020-07-14 21:50:21,814 WARN [kvm.storage.IscsiAdmStorageAdaptor] (Thread-6:null) (logid:) Unable to retrieve the size of device (resource may have moved to a different host)/dev/disk/by-path/ip-192.168.10.10:3260-iscsi-iqn.2001-05.com.equallogic:0-1cb196-21b284232-40d0019858b5679f-cloudstack1-lun-0
At this point all the luns have been cleared out, even though the ISCSI luns were not being managed by CloudStack.
@skattoju4 @GabrielBrascher