Mellanox 网卡使用SR-IOV
from–https://www.cnblogs.com/juzib/p/12895887.html

Mellanox 网卡使用SR-IOV

一、硬件环境准备

物理服务器为:浪潮

网卡型号:Mellanox Technologies MT27800

1、进入BIOS

浪潮服务器为例,开机画面中按“DEL”键;

 

等待系统进入BIOS 后再进行下一步;

成功进入BIOS的画面

 

 

 

2、主板开启SR-IOV

移动光标到“Advanced”标签页面,再选择“PCI Subsystem Settings”,选中按回车;

    再将SR-IOV支持开关设置为“Enabled”;

 

 

 

 

 

 

 

处理器开启虚拟化支持;

将光标移动到“Processor”标签页,再选中“Processor Configration”;

将“VMX”与“SMX”的值设置为“Enabled”;

 

 

 

 

 

再从”Processor ”下找到“IIO Configuration”,进入后将“VT-d”功能打开;

 

 

 

 

操作系统开启iommu

(Centos8.1 为例)

[root@localhost ~]# vim /etc/default/grub

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR=”(seds,release.(���′�,�������.∗,,g’ /etc/system-release)”

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT=”console”

GRUB_CMDLINE_LINUX=”crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet intel_iommu=on iommu=pt”

GRUB_DISABLE_RECOVERY=”true”

GRUB_ENABLE_BLSCFG=true

[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

开启完成后需要重启机器;

驱动环境准备

1、取得下载合适的安装包进行驱动的安装

这里以MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz 为例:

2、解压

[root@localhost ~]# tar xzvf MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz

1、查看readme 安装相应的依赖包:

[root@localhost ~]#yum install perl pciutils python gcc-gfortran libxml2-python tcsh libnl.i686 libnl expat glib2 tcl libstdc++ bc tk gtk2 atk cairo numactl pkgconfig ethtool

2、驱动安装

[root@localhost ~]#./mlnxofedinstall

    3、安装完成后重启机器;

3、配置opensm开启虚拟化

[root@localhost docs]# vim /etc/opensm/opensm.conf

添加内容后保存退出:

virt_enabled 2

参数的解释:

parameter has the following configuration options:

 

0: Ignore Virtualizations – No virtualization support

1: Disable Virtualization – Disable virtualization on all Virtualization supporting ports

2: Enable Virtualization – Enable (virtualization on all Virtualization supporting ports)

4、系统上开启SRIOV

[root@localhost docs] # mst start

Starting MST (Mellanox Software Tools) driver set

Loading MST PCI module – Success

Loading MST PCI configuration module – Success

Create devices

 

[root@localhost docs]# mst status

MST modules:

————

MST PCI module is not loaded

MST PCI configuration module loaded

 

MST devices:

————

/dev/mst/mt4119_pciconf0         – PCI configuration cycles access.

domain:bus:dev.fn=0000:af:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1

Chip revision is: 00

5、查询设备状态信息:

[root@localhost docs]# mlxconfig -d /dev/mst/mt4119_pciconf0 q

 

Device #1:

———-

 

Device type:    ConnectX5

Name:           MCX516A-CCA_Ax

Description:    ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6

Device:         /dev/mst/mt4119_pciconf0

Configurations:                              Next Boot

MEMIC_BAR_SIZE                      0

MEMIC_SIZE_LIMIT                    _256KB(1)

HOST_CHAINING_MODE                  DISABLED(0)

HOST_CHAINING_DESCRIPTORS           Array[0..7]

HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]

FLEX_PARSER_PROFILE_ENABLE          0

FLEX_IPV4_OVER_VXLAN_PORT           0

ROCE_NEXT_PROTOCOL                  254

ESWITCH_HAIRPIN_DESCRIPTORS         Array[0..7]

ESWITCH_HAIRPIN_TOT_BUFFER_SIZE     Array[0..7]

PF_BAR2_SIZE                        0

NON_PREFETCHABLE_PF_BAR             False(0)

VF_VPD_ENABLE                       False(0)

STRICT_VF_MSIX_NUM                  False(0)

VF_NODNIC_ENABLE                    False(0)

NUM_OF_VFS                          4

PF_BAR2_ENABLE                      False(0)

SRIOV_EN                            True(1)

PF_LOG_BAR_SIZE                     5

VF_LOG_BAR_SIZE                     1

NUM_PF_MSIX                         63

NUM_VF_MSIX                         11

INT_LOG_MAX_PAYLOAD_SIZE            AUTOMATIC(0)

上面的两个参数值需要注意,一个是虚拟机出来的VF数量,一个是开启sriov 功能的开关;

如果需要调整请使用:

mlxconfig -d /dev/mst/mt41139pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4

Device #1:

———-

Device type: ConnectI

PCI device: /dev/mst/mt4119_pciconf0

Configurations: Current New

SRIOV_EN 0 1

NUM_OF_VFS 0 4

FPP_EN 0 1

INT_LOG_MAX_PAYLOAD_SIZE 0 0

Apply new Configuration? ? (y/n) [n] : y

Applying… Done!

-I- Please reboot machine to load new configurations.

设置完成后需要重启机器;

6、适配设备

[root@localhost docs]# mlxfwreset –device /dev/mst/mt4119_pciconf0 reset

 

Minimal reset level for device, /dev/mst/mt4119_pciconf0:

 

3: Driver restart and PCI reset

Continue with reset?[y/N] y

-I- Sending Reset Command To Fw             -Done

-I- Stopping Driver                         -Done

-I- Resetting PCI                           -Done

-I- Starting Driver                         -Done

-I- Restarting MST                          -Done

-I- FW was loaded successfully

7、设置网卡的vf参数值.

[root@localhost docs]# echo 4 >/sys/class/net/ens47f1/device/sriov_numvfs

8、检查是否成功;

常用命令检查即可,如ifconfig .ip a ,lspci等;

上一篇
下一篇