Using command line NVMe-oF DEMO to test off-load NVMe-oF target performance

From the package of us, there are few demo utils to create a single pooled storage to demonstrate storage features and performance. Through the way, you can quick preview your favorite features in minutes.

For NVMe-oF target, you may run the following command to start:

/opt/flexsds/bin/flexsds -d nvmf spool 3 nvme://0000:04:00.0 nvme://0000:05:00.0 nvme://0000:05:00.0
outputs are:
found 1 ib devices
device0: mlx5_0, uverbs0, /sys/class/infiniband_verbs/uverbs0, /sys/class/infiniband/mlx5_0
copies: 2
create name space:…done.
NVMe over Fabric service is started.

Using built-in NVMe over Fabric initiator in Linux

Install the nvmf-cli package:

Cent OS: yum install -y nvme-cli
Ubuntu: apt-get install nvme-cli

load the kernel modules:

modprobe nvmet
modprobe nvmet-rdma
modprobe nvme-rdma


Discover and connect:

nvme discover -t rdma -a -s 4420
nvme connect -t rdma -n -a -s 4420
Then you will able to test performance on the new created NVMe device.
Note: The virtual NVMe device is created by the Linux built-in kernel mode NVMf driver, the performance will be limited on this, will not represent to the server’s throughput.

Using built-in NVMf initiator in Linux

We can use user mode perf tool (third-party one) to check performance of the NVMf target:

perf -q 64 -s 4096 -w randrw -M 100 -t 30 -r ‘trtype:RDMA adrfam:IPv4 traddr: trsvcid:4420’ -l -c 0x2

Testing environment and results

CPU: E52680

Memory: 64G

NVMe: Intel NVMe 4610 * 3.

NIC:  Mellanox ConnectX-4 100GB

PERFORMANCE: around 2170,000 IOPS in Read, 790,000 IOPS in Write.