Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 33 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ Runtime - (run on the SoC)


- A Server with the following components
- >16 GB DRAM
- >8 CPU cores
- 16+ GB DRAM
- 8+ CPU cores
- an NVMe SSD
- Broadcom StingRay SVK Board or a RDMA NIC (more details below)

Expand All @@ -114,22 +114,27 @@ Runtime - (run on the SoC)

0. Clone the repo

```
$ git clone https://github.com/huaicheng/LeapIO.git
```

1. Compile host kernel

```
$ cd LeapIO/Linux
$ cp configs/config-rt .config
$ make oldconfig
$ make -j16
$ sudo make modules_install
$ sudo make install
$ sudo update-grub2
```

Configure host grub file to boot newly compiled host kernel

Doule check the line starting with ``GRUB_CMDLINE_LINUX`` contains the following options in ``/etc/default/grub``:

```
GRUB_CMDLINE_LINUX="intel_iommu=off transparent_hugepage=never modprobe.blacklist=nvme,nvme_core"


Expand All @@ -140,30 +145,32 @@ Doule check the line starting with ``GRUB_CMDLINE_LINUX`` contains the following

# Login again after the host is up
$ uname -a # should give output containing "4.15.0-rc4-Leap"

```

Note this step is necessary for successfully building LeapIO Driver in the next
step.


```
# Verify the SSD
sudo modprobe nvme
sudo nvme list
sudo nvme id-ctrl /dev/nvm0n1 | grep -i mdts

```

2. Compile LeapIO WPT kernel module

```
$ cd LeapIO/Driver
$ make
# there should be a kernel module named “wpt.ko” generated, we will load it later
# ls wpt.ko

```

3. Compile FEMU

Note: please change ``bs_size`` in ``hw/block/nvme.c`` to make it same as the value you get from ``sudo blockdev --getsz64 /dev/nvme0n1``

```
$ cd LeapIO/FEMU
$ mkdir -p build-femu
# install dependencies for FEMU
Expand All @@ -176,11 +183,11 @@ Note: please change ``bs_size`` in ``hw/block/nvme.c`` to make it same as the va

The FEMU binary will appear as "x86_64-softmmu/qemu-system-x86_64"
$ ls x86_64-softmmu/qemu-system-x86_64

```

4. Prepare VM images to serve as the user VM and the emulated SoC (referred to as "SoC-VM", it is needed when Broadcom SVK SoC is not available):


```
# Download Ubuntu 16.04 server ISO file
$ wget http://releases.ubuntu.com/16.04/ubuntu-16.04.5-server-amd64.iso
$ sudo apt-get install qemu-system-x86
Expand All @@ -196,70 +203,73 @@ Note: please change ``bs_size`` in ``hw/block/nvme.c`` to make it same as the va
$ sudo shutdown -h now

# Now, socvm.qcow2 is ready to be used with FEMU

```

5. Compile LeapIO runtime (note this step needs to be done inside the VMs, refer back to it later after the VMs are up)


Inside SoC-VM, we need to run ``socp``, first scp it from vSSD/soc-prog into SoC-VM if not, then

```
vm$ cd runtime
vm$ make # the generated binary is "socp"

```

### Run LeapIO

- On the host side, open a separate terminal to observe kernel log outputs, and do the following:

```
$ sudo mknod /dev/wpt c 200 0
$ sudo modprobe nvme
$ cd vSSD/one
$ sudo insmod wpt.ko




$ sudo touch /usr/local/etc/qemu/bridge.conf
\# configure bridge rule to allow traffic through "fbr0", needs to be done once
$ echo "allow fbr0" | sudo tee /usr/local/etc/qemu/bridge.conf

```

- Boot SoCVM:


```
$ cd FEMU/build-femu
$ sudo ./socvm.sh

```

Login to SoC-VM via ssh:

```
$ ssh -p8089 user@localhost

```

Once you're inside the socvm,


```
socvm$ lspci | grep Inter-VM | awk '{print $1}' # this give you the correct ID, modify 84 & 85 accordingly

```

Then, copy ``Runtime/`` to the SoC-VM and compile socp again and run it


```
socvm$ make
socvm$ sudo ./socp # Keep it running in the background

```

- Boot user VM (Referred to as ``DBVM``)

On host machine, do

```
$ cd FEMU/build-femu
$ sudo ./dbvm.sh
# Login to DBVM:
$ ssh -p8080 user@localhost

```

Inside DBVM:

```
dbvm$ sudo modprobe nvme
# Load pblk module if using OCSSD
dbvm$ sudo modprobe pblk
Expand All @@ -271,4 +281,4 @@ Inside DBVM:

# first, modify t.fio filename option to use "filename=/dev/nvme0n1"
dbvm$ sudo fio t.fio

```