misc test log

-hda, vs virtio-blk-pci

Tue Dec 2 18:34:24 JST 2014

for K2:

stderr: real 0m27.664s stderr: real 0m27.698s stderr: real 0m25.832s virtio-blk-pci stderr: real 0m25.780s virtio-blk-pci stderr: real 0m27.680s

for K4:

stderr: real 0m54.959s stderr: real 0m47.808s virtio-blk-pci stderr: real 0m47.842s virtio-blk-pci stderr: real 0m56.563s

data-plane (=3) has no sig difference for boot test

Tried for K2,K3, K4. It does seem to be supported in 0.12.1.

sudo sysctl kernel.sched_compat_yield=1 (on K3)

no difference for K4 boot.

deadline scheduler on K3, no difference

deadline in K3 and noop in K4 made no difference

Plan A

Generate centos.pem for ssh access to all 1box’s created.

Make a OpenVZ 1box (w/ 5GB of disk), and package

Make a nested KVM 1box (w/ 25GB of disk)

Copy OpenVZ 1box and windows image into KVM 1box,

but don’t bother to register Wakame’s images.

Plan A commands

setup

git clone https://github.com/axsh/vmapp-vdc-1box && echo ,,,,, && \ cd vmapp-vdc-1box/ && echo ,,,,, && \ make && echo ,,,,, && \ ./prepare-vmimage.sh kvm i686 centos-6.4 && echo ,,,,, && \ ./prepare-vmimage.sh kvm x86_64 centos-6.4 && echo ,,,,, && \ ./prepare-vmimage.sh kvm x86_64 lb-centos6-stud && echo ,,,,, && \ ./prepare-vmimage.sh kvm x86_64 lbnode && echo ,,,,, && \ ./prepare-vmimage.sh kvm x86_64 vanilla && \ echo ,,,,, && \ ./prepare-vmimage.sh openvz i686 centos-6.4 && echo ,,,,, && \ ./prepare-vmimage.sh openvz x86_64 centos-6.4 && echo ,,,,, && \ ./prepare-vmimage.sh openvz x86_64 lb-centos6-stud && echo ,,,,, && \ ./prepare-vmimage.sh openvz x86_64 lbnode && echo ,,,,, && \ ./prepare-vmimage.sh openvz x86_64 vanilla && echo ,,,,, && \ yes | ssh-keygen -N “” -f centos.pem -C “centos@1box”

build openvz box

sed -i ‘s/((10/((5/’ disk.conf sed -i ‘s/((25/((5/’ disk.conf echo ; cat disk.conf ; sleep 10

time sudo ./box-ctl.sh build openvz

build kvm box

sed -i ‘s/((10/((25/’ disk.conf sed -i ‘s/((5/((25/’ disk.conf echo ; cat disk.conf ; sleep 10

time sudo ./box-ctl.sh build kvm

optimizations

data-plane

http://blog.vmsplice.net/2013/03/new-in-qemu-14-high-performance-virtio.html

If you do not use libvirt the QEMU command-line is:

qemu -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=path/to/disk.img \ -device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

using this: -device virtio-blk,drive=centos-drive,scsi=off,config-wce=off,x-data-plane=on

qtree output

And yes, it appears I am in fact using virtio-blk-data-plane (see snippet from my info qtree output below):

dev: virtio-blk-pci, id “virtio-disk0” class = 0x0 ioeventfd = on vectors = 2 x-data-plane = on

scheduler

there exists a (per) process scheduler option

http://doc.opensuse.org/products/draft/SLES/SLES-tuning_sd_draft/cha.tuning.taskscheduler.html

To retrieve the real-time attributes of an existing task:

saturn.example.com:~ # chrt -p 16244 pid 16244’s current scheduling policy: SCHED_OTHER pid 16244’s current scheduling priority: 0

hmmm

sched_compat_yield Enables the aggressive yield behavior of the old 0(1) scheduler. Java applications that use synchronization extensively perform better with this value set to 1. Only use it when you see a drop in performance. The default value is 0.

Expect applications that depend on the sched_yield() syscall behavior to perform better with the value set to 1.

how to set the I/O scheduler

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/block/switching-sched.txt?id=HEAD

echo ‘sudo bash -c “echo deadline >/sys/devices/pci0000:00/0000:00:04.0/virtio1/block/vda/queue/scheduler”’ | ./nested-kvm-ctrl.sh 3 -doscript bash

in grub.conf

Grub command line – elevator=deadline/cfq/noop

strategies

do fork test

repeat tests on simpler hardware

reference

interesting optimization of KVM involving NUMA and nodes

http://www.linux-kvm.org/wiki/images/5/53/2011-forum-Improving-out-of-box-performance-v1.4.pdf

google: “numa balancing kvm”

http://events.linuxfoundation.jp/sites/events/files/slides/LCJ2014-How_NUMA_Balancing_Moves_KVM_Guests.pdf

http://rhsummit.files.wordpress.com/2014/04/shak-larry-jeder-perf-and-tuning-summit14-part1-final.pdf

How to manage NUMA manually •Research NUMA topology of each system •Make a resource plan for each system •Bind both CPUs and Memory • Might also consider devices and IRQs •Use numactl for native jobs: •“numactl -N <nodes> -m <nodes> <workload>” •Use numatune for libvirt started guests •Edit xml: <numatune> <memory mode=”strict” nodeset=”1-2”/>

concise summary of recommendations

http://drup.org/kvm-disk-performance

The optimal configuration is (usually) as follows:

On the host, set elevator=deadline Use virtio and only virtio use raw LVs whenever possible. Qcow2 gives overhead. Files on a FS also have overhead in the VM use the elevator=noop (See blog.bodhizazen.net/…rformance) both in host and VM, use noatime,nodiratime in fstab wherever possible Make sure the virtio drivers are up to date, especially the windows ones.

http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/wednesday/wagner_w_420_kvm_performance_improvements_and_optimizations.pdf

2014 KVM Forum

https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA

Nested Virtualization by Bandan Das & Jan Kiszska

https://www.youtube.com/watch?v=GvenpiRc1Ac

https://www.youtube.com/watch?v=GvenpiRc1Ac#t=2755

slides: https://drive.google.com/file/d/0BzyAwvVlQckeTng2UmlyamVZVkE/view

scratch

vmbuilder

copy and pasted into shell window:

sed -i ‘s/((10/((25/’ disk.conf

time sudo strace 2>>/tmp/st.log -f -e trace=execve -s 999 ./box-ctl.sh build kvm

result:

it worked: /ssh:triggers@192.168.2.24: #$ pwd ; ls -lsh raw /home/triggers/dev/nested-kvm-performance-experiments/vmapp-vdc-1box 3.4G -rw-r–r– 1 root root 25G Nov 21 13:57 1box-kvm.netfilter.x86_64.raw

there is something for auto insertion of sshkey

vmapp-vdc-1box/vmspec.conf:

ssh_user_key=${BASH_SOURCE[0]%/*}/${devel_user}.pem.pub [[ -f ${BASH_SOURCE[0]%/*}/${devel_user}.pem.pub ]] || ssh_user_key=

strace summary:

[pid 13523] execve(“/bin/bash”, [“/bin/bash”, “-c”, “time sudo VDC_HYPERVISOR=kvm VDC_EDGE_NETWORKING=netfilter setarch x86_64 ./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh build –config-path=./vmbuilder.conf”], [/* 24 vars /]) = 0 [pid 13524] execve(“/bin/sudo”, [“sudo”, “VDC_HYPERVISOR=kvm”, “VDC_EDGE_NETWORKING=netfilter”, “setarch”, “x86_64”, “./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, “build”, “–config-path=./vmbuilder.conf”], [/ 24 vars /]) = 0 [pid 13525] execve(“/bin/setarch”, [“setarch”, “x86_64”, “./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, “build”, “–config-path=./vmbuilder.conf”], [/ 20 vars /]) = 0 [pid 13525] execve(“./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, [“./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, “build”, “–config-path=./vmbuilder.conf”], [/ 20 vars /]) = 0 [pid 13551] execve(“/home/triggers/dev/nested-kvm-performance-experiments/vmapp-vdc-1box/vmbuilder/kvm/rhel/6/misc/../vmbuilder.sh”, [“/home/triggers/dev/nested-kvm-performance-experiments/vmapp-vdc-1box/vmbuilder/kvm/rhel/6/misc/../vmbuilder.sh”, “–config-path=./vmbuilder.conf”], [/ 23 vars */]) = 0

((where is ./vmbuilder.conf ???)) ans: right at base of vmapp-vdc-1box.

another view

execve(“./box-ctl.sh”, [“./box-ctl.sh”, “build”, “kvm”], [/* 20 vars /]) = 0 [pid 13524] execve(“/bin/sudo”, [“sudo”, “VDC_HYPERVISOR=kvm”, “VDC_EDGE_NETWORKING=netfilter”, “setarch”, “x86_64”, “./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, “build”, “–config-path=./vmbuilder.conf”], [/ 24 vars /]) = 0 [pid 13525] execve(“/bin/setarch”, [“setarch”, “x86_64”, “./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, “build”, “–config-path=./vmbuilder.conf”], [/ 20 vars /]) = 0 [pid 13525] execve(“./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, [“./vmbuilder/kvm/rhel/6/misc/kvm-ctl.sh”, “build”, “–config-path=./vmbuilder.conf”], [/ 20 vars /]) = 0 [pid 13551] execve(“/home/triggers/dev/nested-kvm-performance-experiments/vmapp-vdc-1box/vmbuilder/kvm/rhel/6/misc/../vmbuilder.sh”, [“/home/triggers/dev/nested-kvm-performance-experiments/vmapp-vdc-1box/vmbuilder/kvm/rhel/6/misc/../vmbuilder.sh”, “–config-path=./vmbuilder.conf”], [/ 23 vars */]) = 0

(macnotes2) Thursday, November 13th

Just built 60G image with these commands:

ssh:triggers@192.168.2.24: #$ git clone https://github.com/axsh/vmapp-vdc-1box /ssh:triggers@192.168.2.24: #$ cd vmapp-vdc-1box /ssh:triggers@192.168.2.24: #$ make /ssh:triggers@192.168.2.24: #$ ./prepare-vmimage.sh kvm i686 centos-6.4 /ssh:triggers@192.168.2.24: #$ ./prepare-vmimage.sh kvm x86_64 centos-6.4 /ssh:triggers@192.168.2.24: #$ ./prepare-vmimage.sh kvm x86_64 lb-centos6-stud /ssh:triggers@192.168.2.24: #$ ./prepare-vmimage.sh kvm x86_64 lbnode /ssh:triggers@192.168.2.24: #$ ./prepare-vmimage.sh kvm x86_64 vanilla /ssh:triggers@192.168.2.24: #$ cat disk.conf rootsize=${rootsize:-$((60 * 1024))} swapsize=${swapsize:-0} optsize=${optsize:-0} /ssh:triggers@192.168.2.24: #$ time bash -x ./box-ctl.sh build kvm [INFO] Creating disk image: “/home/triggers/dev/fresh-1box-kvm/vmapp-vdc-1box/1box-kvm.netfilter.x86_64.raw” of size: 61440MB [INFO] Generated => /home/triggers/dev/fresh-1box-kvm/vmapp-vdc-1box/1box-kvm.netfilter.x86_64.raw real 8m47.369s user 2m19.799s sys 0m17.325s /ssh:triggers@192.168.2.24: #$ ls -lsh raw 3.4G -rw-r–r– 1 triggers triggers 60G Nov 13 19:22 1box-kvm.netfilter.x86_64.raw

FilesExpand file tree

nested-kvm-notes.org

Latest commit

History