NAS Backup fails when VM Instance is turned on

[kladderadeng]

problem

Testing the NAS Backup feature in 4.20 at the moment.

Host:
Ubuntu 24.04
Apache Cloudstack 4.20.0.0 installed via the ACS Quickstart guide, so not compiled from source here!
Hypervisor KVM QEMU emulator version 8.2.2
Libvirtd v 10.0.0

Guest:
Windows Server 2022 ENU
VirtIO Drivers / Guest Tools intalled in version 0.1.266-1

When the VM Instance is turned off, I can trigger a manual backup without an issue. It backs up to a Backup storage on a TrueNAS Scale instance, connection is via NFS.

When the VM instance is turned on, I get an error message stating that the Backup failed on the ACS GUI. In /var/log/syslog, I get this:

2025-03-01T17:58:06.571961+00:00 delle libvirtd[994]: unable to stat: /tmp/csbackup.rwEm9/i-2-10-VM/2025.03.01.17.58.06/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T17:58:06.572041+00:00 delle libvirtd[994]: Unable to restore security label on /tmp/csbackup.rwEm9/i-2-10-VM/2025.03.01.17.58.06/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2
2025-03-01T17:58:53.288822+00:00 delle libvirtd[994]: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2': Permission denied
2025-03-01T17:58:53.291118+00:00 delle libvirtd[994]: Path '/tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2' is not accessible: No such file or directory
2025-03-01T17:58:53.291218+00:00 delle libvirtd[994]: Unable to tear down cgroup access on /tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2
2025-03-01T17:58:53.294544+00:00 delle libvirtd[994]: internal error: child reported (status=125): unable to stat: /tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T17:58:53.294647+00:00 delle libvirtd[994]: unable to stat: /tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T17:58:53.297680+00:00 delle libvirtd[994]: internal error: child reported (status=125): unable to stat: /tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T17:58:53.297796+00:00 delle libvirtd[994]: unable to stat: /tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T17:58:53.297871+00:00 delle libvirtd[994]: Unable to restore security label on /tmp/csbackup.aJ7Nx/i-2-10-VM/2025.03.01.17.58.52/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2

2nd attempt a bit later looks pretty much the same:

2025-03-01T19:46:40.135152+00:00 delle libvirtd[994]: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2': Permission denied
2025-03-01T19:46:40.136969+00:00 delle libvirtd[994]: Path '/tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2' is not accessible: No such file or directory
2025-03-01T19:46:40.137077+00:00 delle libvirtd[994]: Unable to tear down cgroup access on /tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2
2025-03-01T19:46:40.140287+00:00 delle libvirtd[994]: internal error: child reported (status=125): unable to stat: /tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T19:46:40.140409+00:00 delle libvirtd[994]: unable to stat: /tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T19:46:40.143245+00:00 delle libvirtd[994]: internal error: child reported (status=125): unable to stat: /tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T19:46:40.143333+00:00 delle libvirtd[994]: unable to stat: /tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2: No such file or directory
2025-03-01T19:46:40.143382+00:00 delle libvirtd[994]: Unable to restore security label on /tmp/csbackup.v5VyJ/i-2-10-VM/2025.03.01.19.46.39/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2

versions

OS Ubuntu 24.04
Apache Cloudstack 4.20.0.0 installed via the ACS Quickstart guide, so not compiled from source here!
Hypervisor KVM QEMU emulator version 8.2.2
Libvirtd v 10.0.0

The steps to reproduce the bug

1. Create a Backup repository using NFS
2. Creat a Backup offering
3. Assign the Backup offering to a Windows Server 2022 ENU VM, with the VirtIO drivers / guest tools installed
4. Turn on the VM, wait until all Services are started
5. Select the VM Instance and click on Start Backup to trigger a Backup job

What to do about it?

Backup works fine when the VM is off, so I suppose it has nothing to do with the Backup storage hosted on TrueNAS scale via NFS.
However, the Log messages do not make much sense to me either, claiming that there is an issue accessing files in /tmp. Why should this be different than from taking an Offline Backup? Or is it?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[rajujith]

@kladderadeng The NAS Backup plugin uses different methods for backing up Running and Stopped instances [1]. Have you tried backing up Running Linux instances?

[1] https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/nas_plugin.html#nas-backup-and-recovery-plugin

[abh1sar]

@kladderadeng can you please check if the /tmp/csbackup.rwEm9/i-2-10-VM/2025.03.01.17.58.06/root.dda8b221-33ea-4fd4-b8df-f8b2fc3b96b4.qcow2 file is existing on the NAS server? If yes, what is the owner and permission on the file?

You can try mounting the nas repository on the kvm host manually and giving the virsh backup command and share the result.
For example,

virsh list
 Id   Name        State
---------------------------
 1    s-1-VM      running
 2    v-2-VM      running
 3    r-4-VM      running
 4    i-2-12-VM   running

virsh domblklist i-2-12-VM
 Target   Source
------------------------------------------------------------------------------------------
 vda      /mnt/1fafa00c-f655-3fe9-9cb4-31765495245f/14ce38cd-0522-4172-b880-248d08cabeea
 hdc      -

virsh backup-begin <vm_name> --backupxml backup.xml

backup.xml
<backup>
  <disk name='vda' type='file'>
    <source file='/mnt/1fafa00c-f655-3fe9-9cb4-31765495245f/14ce38cd-0522-4172-b880-248d08cabeea'/>
    <target file='/mnt/nas-repo/root-backup,qcow2'/>
  </disk>
</backup>

[abh1sar]

@kladderadeng can you also share what export flags you are using for the nas backup repository.

[kladderadeng]

Thanks everyone and sorry for the delay. I had to reinstall the whole environment because the system used is undergoing tests with various Hypervisors. But I am back to ACS now and am able to replicate the issue.

@rajujith Thank you. I understand now that the process is different. I also read that this feature is probably not the right one to prepare for a disaster recovery, so we will have to look for other solutions.

@abh1sar Thank you for the detailed step by step explanation. That helps a lot.
Regarding your question about the qcow2 file in the /tmp folder: There is none created. The folder and in it a backup.xml is created, though.

The auto-created backup.xml looks like this:

<domainbackup mode='push'><disks>
<disk name='sda' backup='yes' type='file' backupmode='full'><driver type='qcow2'/><target file='/tmp/csbackup.jlJQw/i-2-16-VM/2025.03.16.07.05.01/root.1b677e62-95d0-43b2-be8d-d7d1919c950f.qcow2' /></disk>
</disks></domainbackup>

Permissions are this:

7 -rw-r--r-- 1 nobody root 263 Mar 16 07:05 /tmp/csbackup.jlJQw/i-2-16-VM/2025.03.16.07.05.01/backup.xml

About manual testing: Since the structure of your sample.xml gave me an error that it does not fit to the schema, I tried to create my own with the help of https://libvirt.org/formatbackup.html, but failed too (skill issue, I am pretty sure ;-) ). I tried to use the one that ACS / virsh auto-created in the /tmp folder and as expected, this replicates the error I found in the logs:

root:~# virsh backup-begin i-2-16-VM --backupxml /tmp/csbackup.jlJQw/i-2-16-VM/2025.03.16.07.05.01/backup.xml
error: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/tmp/csbackup.jlJQw/i-2-16-VM/2025.03.16.07.05.01/root.1b677e62-95d0-43b2-be8d-d7d1919c950f.qcow2': Permission denied

I ruled out "true" permission issues when I created this issue, but after @rajujith pointed out that the backup procedure of turned on / off instances is different, I wonder if changing something on the permission side would help.

I did not add any mount options when mounting the NFS share while creating the Backup Repository in ACS, though. So I went back to my NAS NFS share and added a maproot --> root to see what it does. And now the Live backup started.

It seems I was able to fix it by adjusting the default permissions for the NFS share on the NAS side. What I do not get is, why is the offline backup working and the online backup now. There is a different way how the NFS share is mounted, right?

[abh1sar]

@kladderadeng NFS share is mounted the same way in offline and online backup. I think the difference is that qemu blockdev-add (started by virsh backup-begin) during online backup might be running as a different user, while offline backup does qemu-img convert which runs as root.
if the backup directory has the owner nobody, it might fail for the qemu user, which might be the reason for the error.
Best practice would be to set the owner of the backup directory as root and use maproot->root or no_root_squash.

[kladderadeng]

Thanks everyone for your replies. By fixing the NFS permissions, Offline and Online Backups are working.
@abh1sar maproot -> root is what I used and it is working fine. Thank you :-)