This document describes how to configure and troubleshoot IB switch modules (BX1020B) and IB network adapters (IB1040i) in H3C Uniserver B16000 blade server chassis.

Administrators can connect the IB switch modules to the IB network adapters for management because the IB switch modules have no management port. The IB switch modules and the blade servers with the IB network adapters must be installed into the chassis before the IB switch modules in the Linux operating system for the blade server can be managed (the server with the Redhat 7.5 system installed is taken for example here). A cluster (the whole chassis including the IB network adapters and the IB switch modules) can manage all IB switch modules by using one blade server that meets the requirements.

Chassis IB configuration allows users to uniformly manage the IB devices (referring to IB switch modules and IB network adapters in this document) in the current cluster by enabling the Subnet Manager (SM, which is elaborated in the following chapters) based on the concept of IB clustering. The IB cluster can be divided into a physical layer and a logical layer. The physical layer is in a linkup state when the IB switch modules and the IB network adapters are connected to the midplane of the chassis. The logical layer can work normally only when SM is enabled.

Prerequisites

This chapter describes how to configure SM. SM, which operates on one blade server with the IB network adapter installed, manages all devices in the IB cluster uniformly. Before configuring SM, you must upgrade the IB network adapter driver on the current server, and install the MFT toolkit. MFT is a set of firmware management tools for producing standardized or customized Mellanox firmware images, querying firmware information, and downloading firmware images. To ensure the normal use of SM, the physical connection between the IB switch modules and the IB network adapters must be proper. IP addresses can be set for IB network adapter ports according to service requirements.

Download the IB network adapter driver (IB1040i)

Contact H3C Support, and download the IB network adapter driver installation package based on the version match table.

NOTE:

The file name of IB network adapter driver installation package is subject to the actual situation, "MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.5-x86_64.tgz" is taken as an example in this document.

Install the IB network adapter driver

1. Use FTP or SSH tool to upload the driver package to the /home directory of the node server.

2. Decompress the IB network adapters in the current /home path:

Refer to the tar –xvf MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.5-x86_64.tgz command.

3. Go to the MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.5-x86_64/ directory, and execute the ./mlnxofedinstall command for installing the IB network adapter driver.

4. After installing the driver successfully, restart the server.

Configure the network

After installing the IB network adapter and restarting the server, you must configure an IP address for an IB port. The configuration method is the same as the method to configure an IP address for an Ethernet port, including the following steps:

1. Modify the ifcfg-ib0 configuration file in the /etc/sysconfig/network-scripts/ directory of the blade server where the IB network adapter is located, and configure the IP address and subnet mask.

2. After completing the configuration, restart the network service to enable the IP address to take effect, and execute the systemctl restart network.service command.

3. Configure the subnet manager.

An IB network uses the Subnet Manager (SM) to manage network routes. It can operate on the blade server or IB switch module with a management function (this document introduces how SM operates on the server node). SM is enabled to distribute the Local Identifiers (LIDs) to the devices uniformly in the cluster based on the Globally Unique Identifiers (GUIDs). All LIDs are unique, and cannot be recovered in common circumstances.

SM is integrated into the MLNX_OFED (official driver name) driver, and it has been installed by default after the MLNX_OFED driver is installed. SM can operate on one or more server nodes, but only one SM is in the active state to ensure the normal operation of the cluster. When the master SM is in the down state, the slave SM will replace it. The switching interval between the master and slave SMs will not affect the services temporarily. But if SM is disabled, the logical layers of all IB network adapter ports in the cluster will be initialized, which will affect the services.

4. Enable the subnet manager.

Enable the subnet management function on the server node.

· Method 1: Execute the /etc/init.d/opensmd start command.

The active port of the first IB network adapter on the current server is SM management port by default. If the slave SM is needed in the cluster, you can execute the /etc/init.d/opensm start command in other servers of the cluster.

· Method 2: Execute the opensm –B –g <GUID> -p <sm_priority> command to set the priority for the IB network adapter ports. The port, which has the highest priority and is in the active state, is selected as the master SM, and other ports are slave SMs. <GUID> indicates the GUID of the IB network adapter, which can be queried by executing the ibstat command; <sm_priority> indicates the priority of SM in the value range from 0 to 14. The larger the value, the higher the priority.

Download the MFT tool

MFT is a set of firmware management tools for producing standardized or customized Mellanox firmware images, querying firmware information, and downloading firmware images. Download the MFT tool online.

NOTE:

The file name of the MFT toolkit is subject to the actual situation, "mft-4.13.0-104-x86_64-rpm.tgz" is taken as an example in this document.

Install the MFT tool

1. Use the FTP or SSH tool to upload the MFT toolkit to the /home directory of the blade server.

2. Execute the tar -xvf mft-4.13.0-104-x86_64-rpm.tgz command to decompress the general MFT toolkit for IB.

3. Go to the /home/ mft-4.13.0-104-x86_64-rpm / directory, and execute the ./install.sh --oem command for installing the MFT tool.

4. After the MFT tool is installed successfully, execute the mst start command to confirm that the tool has been installed successfully and can run.

Manage IB devices

Set SM to manage all IB devices uniformly in the current cluster.

1. The node server starts the service function of the mst command. Execute the mst start command. Skip if this command has been executed.

2. Automatically add all IB devices in the cluster into SM for uniform management, and execute the mst ib add command.

3. View the IB devices added in the cluster, and execute the mst status –v command (Inband devices in screenshot refer to devices that have been added).

4. You can upgrade and query the current devices after the devices are added to the cluster. See "Upgrade the firmware of IB switch modules and IB network adapters" for details about the usage.

Common commands for IB network adapter (IB1040i)

View the state of a network adapter port

Use this command to view basic information about an IB network adapter.

Syntax

ibstat

Examples

Command output description

CA ‘mlx5_0’: name of network adapter port for additional parameter when the traffic is injected into the specified port.

Firmware version: firmware version of the current network adapter.

State: connection state of network adapter in the logical layer.

Physical state: connection state of network adapter in the physical layer.

Rate: velocity.

Base lid: LID distributed by SM.

SM lid: LID of current SM.

Port GUID: GUID of every network adapter port, which can be used for specifying SM.

View the names and states of ports corresponding to the network adapters

Use this command to view the correspondence between the names of the network adapters in the current system and the ib ports in the system and their states.

Syntax

ibdev2netdev

ibdev2netdev -v

Examples

View all IB network adapter devices in the cluster

Use this command to list all IB network adapter devices in the cluster.

Syntax

ibhosts

Examples

Command output description

0xe41d2d0030570ce6: GUID of the IB network adapter port.

wlp-node8 HCA-1: name of the IB network adapter in the server system.

View the details about the network adapter port

Use this command to view the details about the network adapter port. This command has the similar function to the ibstat command, but has the PSID (board_id) information.

Syntax

ibv_devinfo

Examples

Command output description

Board_id: PSID manufacturer's information code.

View the GUID of the network adapter port in the server

Use this command to view the node GUID of network adapter in the server system.

Syntax

ibv_devices

Examples

View the GUID of the currently running SM

Use this command to query the GUID of SM in the current cluster, indicating that the current SM has been specified.

Syntax

sminfo

Examples

Enable the service function of the mst command

Use this command to enable the service of the mst command in the cluster. This command is used to enable the registration and access procedure, and list the devices that can use the mst command.

Syntax

mst start

Examples

Add the IB devices into the cluster

Use this command to add all IB devices in the cluster into the current cluster for unified management.

Syntax

mst ib add

Examples

Command output description

Added 6 IB-band devices: indicates the number of devices that can be added to the current cluster (including IB switch modules and IB network adapters).

View all devices in the current cluster

Use this command to view all IB devices in the cluster.

Syntax

mst status –v

Examples

Command output description

DEVICE_TYPE: model, full name, PCI number, and associated CPUs of the IB network adapter.

Inband devices: You must convert the LIDs distributed for the hexadecimal devices to decimal ones.

Query the information about IB switch modules and IB network adapter devices

Use this command to manage and use the devices in the current cluster, and query the basic information about the network adapter or the IB switch module.

Syntax

flint -d <IB device> query

Parameters

<IB device>: indicates the name of IB switch module and IB network adapter, which can be queried by executing the mst status –v command, such as the /dev/mst/SW_MT54000_Quantum_Mellanox_Technologies_lid-0x0001 command. The IB switch module is started with SW, and the IB network adapter is started with its name, such as mt4119_pciconf0.

Examples

# Query the IB switch module firmware version

# Query the IB network adapter firmware version.

Upgrade the firmware of IB switch modules and IB network adapters

Use this command to upgrade the firmware of the IB network adapters and the IB switch modules.

Syntax

flint -d <IB device> -i < fw version> burn.

Parameters

< fw version>: indicates the firmware version.

For the detailed parameters, enter flint --help for reference.

Examples

# Power off and restart the IB switch module to take effect after the upgrading of the IB switch module firmware is completed.

# Restart the server to enable the IB network adapter firmware to take effect after the upgrading of the IB network adapter firmware is completed.

Query the port rate and state

Use this command to query the states of internal and external ports of the IB switch module and the state of the IB network adapter port.

Syntax

To view the state of the IB network adapter port:

mlxlink -d <IB device>

To view the state of the port of the IB switch module:

mlxlink -d <IB device> -port <port number>

Parameters

<port number>: indicates the port number in the value range from 1 to 41.

For the detailed parameters, enter mlxlink --help for reference.

Examples

# Query the state and rate of the current IB network adapter port.

# View the state and rate of port 33 of the IB switch module.

Command output description

Speed: indicates the maximum bandwidth rate supported by the current network adapter.

Enabled Link Speed: indicates the negotiation rate supported by the current network adapter.

Support Cable Speed: indicates the negotiation rate supported by the cable that connects the IB switch module (the red font indicates no cable in internal connection).

Set the up or down state of the IB network adapter

Use this command to set the up or down state of the port of the IB network adapter, which is applicable only to the IB network adapter on the current server.

Syntax

mlxlink -d <IB device> -a <up/down>

Parameters

<IB device>: indicates the name of the IB network adapter, which can be queried by executing the mst status –v command, such as the mt4119_pciconf0 command.

<up/down>: indicates the up or down state of the port valuing UP or DN, and DN indicates down.

Examples

# Set the IB network adapter port to be in the down state.

# Set the IB network adapter port to be in the up state.

Common commands for IB switch module (BX1020B)

Query ports of the IB switch module

Use this command to view the connection states of all ports of the IB switch module. This command lists all IB switch modules in the cluster.

Syntax

iblinkinfo

Examples

Command output description

1. Indicates the GUID of this switch, which is a unique identifier for distinguishing it from other switches.

2. LID indicates a unique LID for convenient management distributed by SM when the cluster is created.

3. Indicates that ports 1–20 are internal ports of the IB switch module that connect the network adapters, and ports 11–12 and 17–18 are reserved ports.

4. Indicates that ports 21–40 are external ports of the IB switch module.

5. Indicates that port 41 is a port of the virtual machine, which is a free routed port of the cluster for releasing the pressure.

6. Indicates that the connection rate of the current port is 4X*25.78125=100 Gpbs (EDR).

7. Indicates the LID of the network adapter port and the corresponding server.

8. Indicates the LID of the external port, the serial number of the port, and the name of the peer connection device of the IB switch module.

Query the information about IB switch modules and IB network adapter devices

Use this command to manage and use the devices in the current cluster, and query the basic information about the network adapters or the IB switch modules. See "Query the information about IB switch modules and IB network adapter devices" for more details.

Upgrade the firmware of IB switch modules and IB network adapters

Use this command to upgrade the firmware of the IB network adapters and the IB switch modules. See "Upgrade the firmware of IB switch modules and IB network adapters" for more details.

View the corresponding devices of all IB switch modules in the cluster

Use this command to view all IB switch modules in the cluster.

Syntax

ibnetdiscover

Examples

Command output description

1. Indicates the GUID of the switch for distinguishing the IB switch modules in the cluster.

2. Indicates the information and connection port of the device that connects the current switch.

View the IB switch modules in place

Use this command to view all IB switch modules in the current cluster. This command lists all devices in the cluster one by one.

Syntax

ibswitches

Examples

Command output description

Displays the total ports of the current IB switch module and the LIDs in the cluster.

Set the port of the IB switch module to be in the up or down state

Use this command to change the port of the IB switch module to be in the up or down or reset state.

Syntax

ibportstate <LID> <Port> <port state>

Parameters

<LID>: indicates the LID of the device where the port is located.

<Port>: port number.

<port state>: includes the enable and disable states; enable indicates that the port is in the up state, and disable indicates that the port is in the down state.

For the detailed parameters, enter ibportstate --help for reference.

Examples

# View the state of port 33 of the device with LID as 1.

# Set the state of port 33 of the device with LID as 1 to be in the down state.

# Set the state of port 33 of the device with LID as 1 to be in the up state.

Check the health of optical fiber physical links in the current environment

Use this command to detect the health of the physical links in the current cluster. This command is used to view the states of the links quickly in the environment.

The log information is stored in the /var/tmp/ibdiagnet2/ directory after this command is executed.

Syntax

ibdiagnet

Parameters

For the detailed parameters, enter ibdiagnet --help for reference.

Examples

Collect IB switch module logs

Use this command to collect the log information about the ports of the IB switch module.

Syntax

mlxdump -d <IB device> snapshot --mode full -o <log name>

Parameters

<IB device>: indicates the name of the IB switch module and IB network adapter, which can be queried by executing the mst status –v command.

<log name >: indicates the log file, such as Quantum_mlxdump1.log.

For the detailed parameters, enter flint --help for reference.

Examples

FAQs

IB network adapter

IB network adapter port cannot be displayed

To resolve the issue:

1. Verify if the IB network adapter is compatible with the server.

¡ If a non-standard system is used, contact the corresponding OS R&D Department for solving.

¡ If the IB network adapter version does not match, upgrade it first.

2. Verify if the PCIe hardware device can be displayed (lspci |grep Mellanox).

¡ If the PCIe device cannot be displayed, check if the corresponding CPU is in place, and if the IB network adapter is not installed in place; and replace the slot of the IB network adapter.

¡ If the PCIe device can be displayed, but the network adapter port cannot be displayed, execute the ifconfig –a /ifconfig ibN up command, then reinstall the driver, and reboot the system.

IB port cannot be reached or stays in initialization state

To resolve the issue:

1. Verify if the IB network adapter is up and in a link state.

¡ If yes, use the ifconfig ibN up and ibstat commands to view if the IP is set for the correct port.

¡ If no, it is recommended to use the /etc/init.d/opensm start, /etc/init.d/opensm status command to check whether the function of the opensm command in the cluster is enabled.

2. Replace the slot of the current IB network adapter, and check whether the IB network adapter operates normally.

3. View whether the IB switch module that connects the current network adapter is available.

Rate of the IB network adapter port is negotiated abnormally

To resolve the issue:

1. Verify if the current IB network adapter can support the current rate to be negotiated.

¡ If no, replace the supported IB Mezz network adapter.

¡ If yes, verify if the rate of the port is reduced through negotiation. See "Query the port rate and state" for specific commands.

2. Replace the slot of the current network adapter, and check whether the network adapter operates normally.

Failed to execute the commands for the IB network adapter to query networking device in the system, for example, failed to get IB deviceslist is reported during the execution of the ibv_devinfo command

Call the command of the IB network adapter when the driver is unloaded, and execute the /etc/init.d/openibd start command.

IB switch module

Table 1 Troubleshooting of the IB switch module

Fault type	Symptom	Cause and troubleshooting
LEDs	The port LED of the panel is steady amber	This state indicates that only the physical layer of the port is up: Confirm that SM has operated in the current network: /etc/init.d/opensm status; Replace the current cable, and confirm non-cable problems.
	The port LED of the panel flickers amber	This state indicates that the cable may be faulty: Replace the current cable, and confirm non-cable problems; Replace other ports of the panel, and confirm non-port problems.
	The IB switch module LED flickers red	This state indicates that the switch has an alarm: Confirm that the fan works normally, which does not cause high temperature of the IB switch module;
	The IB switch module system LED is steady off	This state indicates that the IB switch module may not be powered on normally: Confirm that the OM power supply works normally, and can power on the IB switch module normally; Confirm that the IB switch module firmware version and GUID are correct.

H3C UniServer BX1020B IB Switch Module Configuration Guide-6W100

Cloud & AI

InterConnect

Intelligent Terminal Products

Product Support Services

Technical Service Solutions

Resource Center

Policy

Online Help

Become a Partner

Partner Resources

Partner Business Management

Company Information

News & Events

Contact Us