diff --git a/README.md b/README.md index e12672b99..6662aa04d 100644 --- a/README.md +++ b/README.md @@ -104,8 +104,8 @@ Runtime - (run on the SoC) - A Server with the following components - - >16 GB DRAM - - >8 CPU cores + - 16+ GB DRAM + - 8+ CPU cores - an NVMe SSD - Broadcom StingRay SVK Board or a RDMA NIC (more details below) @@ -114,10 +114,13 @@ Runtime - (run on the SoC) 0. Clone the repo +``` $ git clone https://github.com/huaicheng/LeapIO.git +``` 1. Compile host kernel +``` $ cd LeapIO/Linux $ cp configs/config-rt .config $ make oldconfig @@ -125,11 +128,13 @@ Runtime - (run on the SoC) $ sudo make modules_install $ sudo make install $ sudo update-grub2 +``` Configure host grub file to boot newly compiled host kernel Doule check the line starting with ``GRUB_CMDLINE_LINUX`` contains the following options in ``/etc/default/grub``: +``` GRUB_CMDLINE_LINUX="intel_iommu=off transparent_hugepage=never modprobe.blacklist=nvme,nvme_core" @@ -140,30 +145,32 @@ Doule check the line starting with ``GRUB_CMDLINE_LINUX`` contains the following # Login again after the host is up $ uname -a # should give output containing "4.15.0-rc4-Leap" - +``` Note this step is necessary for successfully building LeapIO Driver in the next step. - +``` # Verify the SSD sudo modprobe nvme sudo nvme list sudo nvme id-ctrl /dev/nvm0n1 | grep -i mdts - +``` 2. Compile LeapIO WPT kernel module +``` $ cd LeapIO/Driver $ make # there should be a kernel module named “wpt.ko” generated, we will load it later # ls wpt.ko - +``` 3. Compile FEMU Note: please change ``bs_size`` in ``hw/block/nvme.c`` to make it same as the value you get from ``sudo blockdev --getsz64 /dev/nvme0n1`` +``` $ cd LeapIO/FEMU $ mkdir -p build-femu # install dependencies for FEMU @@ -176,11 +183,11 @@ Note: please change ``bs_size`` in ``hw/block/nvme.c`` to make it same as the va The FEMU binary will appear as "x86_64-softmmu/qemu-system-x86_64" $ ls x86_64-softmmu/qemu-system-x86_64 - +``` 4. Prepare VM images to serve as the user VM and the emulated SoC (referred to as "SoC-VM", it is needed when Broadcom SVK SoC is not available): - +``` # Download Ubuntu 16.04 server ISO file $ wget http://releases.ubuntu.com/16.04/ubuntu-16.04.5-server-amd64.iso $ sudo apt-get install qemu-system-x86 @@ -196,70 +203,73 @@ Note: please change ``bs_size`` in ``hw/block/nvme.c`` to make it same as the va $ sudo shutdown -h now # Now, socvm.qcow2 is ready to be used with FEMU - +``` 5. Compile LeapIO runtime (note this step needs to be done inside the VMs, refer back to it later after the VMs are up) Inside SoC-VM, we need to run ``socp``, first scp it from vSSD/soc-prog into SoC-VM if not, then +``` vm$ cd runtime vm$ make # the generated binary is "socp" - +``` ### Run LeapIO - On the host side, open a separate terminal to observe kernel log outputs, and do the following: +``` $ sudo mknod /dev/wpt c 200 0 $ sudo modprobe nvme $ cd vSSD/one $ sudo insmod wpt.ko - - - + $ sudo touch /usr/local/etc/qemu/bridge.conf \# configure bridge rule to allow traffic through "fbr0", needs to be done once $ echo "allow fbr0" | sudo tee /usr/local/etc/qemu/bridge.conf - +``` - Boot SoCVM: - +``` $ cd FEMU/build-femu $ sudo ./socvm.sh - +``` Login to SoC-VM via ssh: +``` $ ssh -p8089 user@localhost - +``` Once you're inside the socvm, - +``` socvm$ lspci | grep Inter-VM | awk '{print $1}' # this give you the correct ID, modify 84 & 85 accordingly - +``` Then, copy ``Runtime/`` to the SoC-VM and compile socp again and run it - +``` socvm$ make socvm$ sudo ./socp # Keep it running in the background - +``` - Boot user VM (Referred to as ``DBVM``) On host machine, do +``` $ cd FEMU/build-femu $ sudo ./dbvm.sh # Login to DBVM: $ ssh -p8080 user@localhost - +``` Inside DBVM: +``` dbvm$ sudo modprobe nvme # Load pblk module if using OCSSD dbvm$ sudo modprobe pblk @@ -271,4 +281,4 @@ Inside DBVM: # first, modify t.fio filename option to use "filename=/dev/nvme0n1" dbvm$ sudo fio t.fio - +```