Ceph RBD Feature flag update

[Jazz9]

• Improvement Request
  Cloudstack Ceph Integration

When use Ceph >= Jewel / KVM with Cloudstack the RBD feature codes are not the new default values.

SUMMARY

When Cloudstack creates a VM from template it manually specifies the feature set and forces layering only. If you manually clone the volume via the rbd command line tool, the expected features are retained from the parent template.

The features that I require are for these use cases:

1.Using 'rbd du' the get the real disk usage of a volume without fast-diff is very slow. We need to schedule this after hours on large images.
2.We are starting to use rbd mirroring across to our secondary DC and this requires exclusive-lock.
3.Deep flatten can not be turned on after image creation. I am not sure if I require this feature at this time.

STEPS TO REPRODUCE

ACTUAL RESULTS

Example of VM volume from template: - 

user@NAS-AUBUN-RK3-CEPH01:~# rbd info AUBUN-KVM-CLUSTER01-SSD/feeb52ec-f111-4a0d-9785-23aadd7650a5

rbd image 'feeb52ec-f111-4a0d-9785-23aadd7650a5':
        size 150 GB in 38400 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.142926a5ee64
        format: 2
        features: layering
        flags:
        create_timestamp: Fri Apr 27 12:46:21 2018
        parent: AUBUN-KVM-CLUSTER01-SSD/d7dcd9e4-ed55-44ae-9a71-52c9307e53b4@cloudstack-base-snap
        overlap: 150 GB

Note the features are not the same as the parent : -

user@NAS-AUBUN-RK3-CEPH01:~# rbd info AUBUN-KVM-CLUSTER01-SSD/d7dcd9e4-ed55-44ae-9a71-52c9307e53b4
rbd image 'd7dcd9e4-ed55-44ae-9a71-52c9307e53b4':
        size 150 GB in 38400 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.141d274b0dc51
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:
        create_timestamp: Fri Apr 27 12:37:05 2018

@wido

[Jazz9]

Additionally,

Per per dev mailing list discussion with Wido. This following code controls the feature set.

https://github.com/apache/cloudstack/blob/master/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java#L957

private int rbdFeatures = (1 << 0); /* Feature 1<<0 means layering in RBD format 2 */

rbd.clone(template.getName(), rbdTemplateSnapName, io, disk.getName(),
rbdFeatures, rbdOrder);

I would suggest that we create a global setting that controls the feature flags.

1. Default -1 to not specify any flags to rbd.clone. This should choose the defaults for the current version of Ceph.
2. Specify a number the enable that feature code. ( https://ceph.com/geen-categorie/ceph-enable-the-object-map-feature/ )

[wido]

I remember now :) This is leftover code from when libvirt didn't support cloning volumes yet.

Libvirt supports cloning a RBD image: https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/storage/storage_backend_rbd.c;h=642cacb67362424b00e905bab324dad98dd98e8e;hb=HEAD#l945

We should call that function in libvirt to clone the image:

• https://libvirt.org/sources/virshcmdref/html/sect-vol-clone.html
• https://libvirt.org/docs/libvirt-appdev-guide-python/en-US/html/libvirt_application_development_guide_using_python-Storage_Pools-Cloning.html

This should be supported by the libvirt Java bindings as well, so I think we can already do this.

Now somebody just needs to write the code, that's all :)

Cloning is then done by libvirt and CloudStack doesn't need to worry about it.

[Jazz9]

That would be much cleaner. Do you think it is an easy change?

[wido]

@Jazz9 It will need a bit of work in the CloudStack agent as we should make a generic method to have libvirt handle the volume clone. This isn't only required for RBD, but in other cases libvirt can handle this as well.

I took a quick look, but this would require some work.

[wido]

I took a quick look and I think this should work, but, I currently do not have a system to test this on.

https://github.com/wido/cloudstack/commits/rbd-clone-libvirt

wido@035e97b

[wido]

I just found out, libvirt in Ubuntu 16.04 doesn't support this yet, so it will take a long time (years) before we can put this upstream as we have to support Ubuntu 16.04 until at least 2021.

[Jazz9]

O no! How are the new volumes created? They seem to get the correct feature flags. Kind regards, Glen Baars From: Wido den Hollander <notifications@github.com> Sent: Tuesday, 22 May 2018 8:17 PM To: apache/cloudstack <cloudstack@noreply.github.com> Cc: Glen Baars <glen@onsitecomputers.com.au>; Mention <mention@noreply.github.com> Subject: Re: [apache/cloudstack] Ceph RBD Feature flag update (#2650) I just found out, libvirt in Ubuntu 16.04 doesn't support this yet, so it will take a long time (years) before we can put this upstream as we have to support Ubuntu 16.04 until at least 2021. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2650 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AljekbyVPGd6viF2ERR6qbf8G5PfmiJRks5t1AHVgaJpZM4UDPR->. This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

[wido]

@Jazz9 Creating is handled by libvirt, but the cloning function wasn't implemented before Ubuntu 16.04 was released. I didn't check CentOS 7, but I assume the same.

So we will have to statically define the features we want to use. But as we still have people running older Ceph clusters we should be cautious.

Do you want to object-map feature enabled?

[Jazz9]

I think it might be best to leave the default action as it currently is. Should we have a global setting for ceph.clone.rbdfeaturecode?

…

Sent from my phone On 22 May 2018 9:20 PM, Wido den Hollander <notifications@github.com> wrote: @Jazz9<https://github.com/Jazz9> Creating is handled by libvirt, but the cloning function wasn't implemented before Ubuntu 16.04 was released. I didn't check CentOS 7, but I assume the same. So we will have to statically define the features we want to use. But as we still have people running older Ceph clusters we should be cautious. Do you want to object-map feature enabled? - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2650 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AljekTutHX2MJmwTl70GNDNauAw_5o_vks5t1BCXgaJpZM4UDPR->. This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

[wido]

I will leave it there. A global setting does not seem right to me as this isn't something a user should worry about. Not sure yet what to do with this.

[wido]

I might have a solution: https://github.com/wido/cloudstack/commits/rbd-clone-libvirt

Check for the libvirt version and based on that decide to use libvirt or stick with the old code.

Haven't tested this code, but I think it should work just fine.

You need at least libvirt 2.0.0 for this to work

[yadvr]

Looks like this is fixed in #2985, closing this ticket.