Contents

Configuring NetAnalysis· 1

About NetAnalysis· 1

NetAnalysis architecture· 1

NetAnalysis workflow· 2

Configuring NetAnalysis for RoCEv2 traffic· 3

About NetAnalysis for RoCEv2 traffic· 3

RoCEv2 packet format 3

CM connection setup· 4

RoCEv2 flow analysis· 5

RoCEv2 flow aging· 5

RoCEv2 flow filtering· 6

Restrictions: Software version compatibility with NetAnalysis· 6

NetAnalysis for RoCEv2 traffic tasks at a glance· 6

Setting the mode of RoCEv2 traffic analysis· 6

Enabling RoCEv2 traffic statistics collection· 8

Enabling RoCEv2 packet loss analysis globally· 8

Enabling AI ECN for RoCEv2 traffic statistics collection· 9

Setting the interval for reporting RoCEv2 traffic statistics to the NDA· 9

Setting the aging timer for inactive RoCEv2 flows· 9

Display and maintenance commands for NetAnalysis for RoCEv2 traffic· 10

NetAnalysis for RoCEv2 traffic configuration examples· 10

Example: Configuring NetAnalysis to analyze RoCEv2 traffic· 10

 

Configuring NetAnalysis

About NetAnalysis

NetAnalysis is a network traffic monitoring and analysis technology that performs in-depth analysis of service flows to obtain performance statistics about the service flows, such as packet loss rate and latency. NetAnalysis can send the analysis results to a NetAnalysis processor for analysis and display, which helps you monitor the network operation status and locate network faults.

NetAnalysis architecture

A typical NetAnalysis system consists of the following elements:

·     NetAnalysis data exporter (NDE)—Replicates service flows to a NetAnalysis processor.

·     NetAnalysis processor (NAP)—Processes and analyzes service flows and outputs the results to a NetAnalysis data analyzer.

·     NetAnalysis data analyzer (NDA)—Provides a Web interface for obtaining, displaying, and analyzing service flow data.

Typically, an NDE and an NAP are collocated on a device. As shown in Figure 1, when a service flow and its return traffic are forwarded along the same path, all devices on the path obtain bidirectional traffic of the service flow. You can configure NetAnalysis on these devices to analyze the flow to obtain performance statistics such as packet loss rate and latency.

Figure 1 NetAnalysis system

 

NetAnalysis workflow

As shown in Figure 2, NetAnalysis works as follows:

1.     The NDE uses ACLs to match the service flows monitored by NetAnalysis and replicates the service flows to the NAP.

2.     The NAP creates a flow table for each received service flow to analyze it if the service flow satisfies certain requirements. The NAP drops a service flow if the service flow does not meet those requirements or the NAP is overloaded.

3.     The NAP encapsulates the analysis results into packets destined for the NDA, looks up the routing table for a matching route, and sends the packets to the NDA for analysis and display.

Figure 2 NetAnalysis workflow

 

Configuring NetAnalysis for RoCEv2 traffic

About NetAnalysis for RoCEv2 traffic

Remote direct memory access (RDMA) is a direct memory access technology used in InfiniBand networks to resolve the delay of data processing on servers. RDMA transmits data directly through the network from one system to another system without involving either one's operating system. CPUs do not process the data in transmission, which increases the bandwidth and reduces latency and resource usage.

RDMA over converged Ethernet version 2 (RoCEv2) is an RDMA technology used on Ethernet networks. RoCEv2 is widely used to reduce the latency caused by CPU processing and improve application performance in TCP/IP transmission for high-performance computing, distributed storage, and AI. In these scenarios, multiple nodes might send packets simultaneously to the same node, and the burst of traffic will congest queues or even cause packet loss on the destination node. As a result, the network latency increases, and traffic throughput drops. To resolve the issues, configure NetAnalysis to monitor the status of RoCEv2 networks by analyzing RoCEv2 flow data such as packet loss, latency, throughput, and forwarding path.

RoCEv2 packet format

RoCEv2 is a network layer protocol that enables Layer 3 communication between broadcast domains. RoCEv2 encapsulates packets based on the UDP encapsulation. Figure 3 shows the format of an RoCEv2 packet.

Figure 3 RoCEv2 packet format

 

An RoCEv2 packet contains the following fields:

·     Ethernet header—Includes the source and destination MAC addresses.

·     IP header—Includes the source and destination IP addresses.

·     UDP header—Includes the source and destination port numbers. The destination port number is fixed at 4791.

·     InfiniBand base transport header—Includes key fields monitored by NetAnalysis.

·     IB payload.

·     ICRC and FCS.

The InfiniBand base transport header contains the following fields:

·     Opcode—RoCEv2 packet type indicating the operation mode. Available values for this field include the following:

¡     ConnectMsg—The packet is used for setting up an RoCEv2 connection. The connection is called a communication management (CM) connection. Devices in an RoCEv2 network transmit data packets through CM connections.

¡     Send—The packet is sent to the remote end without specifying where the receiver stores data.

¡     Write—The packet carries the address, key, and length of data to be written to the remote end.

¡     Read—The packet carries the address, key, and length of data to be read from the remote end. RoCEv2 packets of the Send, Write, and Read types are analyzed during throughput analysis.

¡     ACK—The packet is a response message returned by the receiver. Based on the ACK extended transport header unique to RoCEv2 ACK packets, an ACK packet can be one of the following types:

-     Common ACK packet indicating that data is received successfully.

-     NAK packet that indicates packet loss.

·     Dest QP—Destination QP that identifies an RoCEv2 flow. This field is similar to the destination port number. It is a key value used by NetAnalysis to create an RoCEv2 flow table.

·     PSN—Sequence number of the RoCEv2 packet. Packet loss is determined by checking whether the PSNs of packets are consecutive. If packet loss occurs, the receiver returns an NAK packet.

CM connection setup

RDMA sets up CM connections based on RoCE packets or TCP packets with custom fields. NetAnalysis can analyze the RoCE packets and TCP packets used for CM connection setup. The analysis process does not differ much between the packet types. The following information uses RoCE packet-based CM connection setup as an example.

Figure 4 shows the process of CM connection setup.

Figure 4 CM connection setup process

 

During CM connection setup, the server and the client send the following RoCEv2 ConnectMsg packets:

·     Connect Request—Sent by the client to the server to request RoCEv2 connection setup.

·     Connect Reply—Returned by the server to the client. After receiving this packet, the client determines that an RoCEv2 connection has been set up with the server.

·     ReadyToUse—Sent by the client to the server. After receiving this packet, the server determines that the CM connection is set up successfully.

RoCEv2 flow analysis

After NetAnalysis is enabled to collect RoCEv2 traffic statistics on the device, the NDE issues rules for matching RoCEv2 packets based on the Opcode field. The NAP creates flow entries to form an RoCEv2 flow table based on the 4-tuple information in RoCEv2 connection setup packets. The 4-tuple information is the IP address of the client, IP address of the server, QP of the client, and QP of the server.

The NAP collects statistics about key fields in the flow table that is created based on the RoCEv2 data packets sent by the NDE, and analyzes the statistics to obtain characteristics of the RoCEv2 flow. You can view the statistics in the flow table on the device, and the statistics are exported to the NDA for display and analysis after the flow ages out.

RoCEv2 flow aging

The RoCEv2 flow aging mechanism allows the device to output flow statistics to the NDA. After NetAnalysis is enabled to analyze RoCEv2 traffic, the device saves flow statistics in the RoCEv2 cache. When an RoCEv2 flow ages out, the device exports the related flow statistics to the NDA and deletes the flow statistics from the RoCEv2 cache to save cache space.

Only inactive RoCEv2 flows age out. The device starts an inactive flow aging timer after receiving a packet for a flow. If the device has not received any packet for the flow when the timer expires, the flow ages out. To save cache space, the device will delete the sessions for inactive flows and notifies the NDA of the deletion events.

RoCEv2 flow filtering

NetAnalysis can use ACLs to filter the RoCEv2 flows that traverse the device. You can use this feature to collect statistics about the RoCEv2 flows of interest. For more information about ACLs, see ACL and QoS Configuration Guide.

Restrictions: Software version compatibility with NetAnalysis

NetAnalysis is supported only in R6635 and later.

NetAnalysis for RoCEv2 traffic tasks at a glance

To configure NetAnalysis to analyze RoCEv2 traffic, perform the following tasks:

1.     Setting the mode of RoCEv2 traffic analysis

2.     Enabling RoCEv2 traffic statistics collection

3.     Enabling RoCEv2 packet loss analysis globally

4.     Enabling AI ECN for RoCEv2 traffic statistics collection

5.     (Optional.) Setting the interval for reporting RoCEv2 traffic statistics to the NDA

6.     (Optional.) Setting the aging timer for inactive RoCEv2 flows

Setting the mode of RoCEv2 traffic analysis

About this task

As shown in Figure 5, NetAnalysis can use either of the following modes for RoCEv2 traffic analysis when multiple paths exist between an RoCEv2 client and an RoCEv2 server:

·     Bidirectional mode—NetAnalysis monitors bidirectional traffic sent between the server and the client. Based on the 4-tuple information in CM connection setup packets, NetAnalysis can collect RoCEv2 traffic statistics based on sessions and provide the session-specific RTT and lost packet count.

·     Unidirectional mode—NetAnalysis monitors the traffic sent from the client to the server to obtain the 3-tuple information used for creating RoCEv2 flow entries. In this mode, NetAnalysis collects RoCEv2 traffic statistics based on flows and provides only packet throughput information.

As a best practice to ensure correct RoCEv2 traffic analysis, configure the mode of RoCEv2 traffic analysis as follows:

·     Enable bidirectional mode on the devices attached to the server and the client (Device C and Device D).

·     Enable unidirectional mode on the intermediate devices (Device A and Device B).

Figure 5 Mode of RoCEv2 traffic analysis

 

Restrictions and guidelines

For NetAnalysis features to take effect, first set the mode of RoCEv2 traffic analysis.

When you change the mode of RoCEv2 traffic analysis, all NetAnalysis configuration and the RoCEv2 cache are cleared. Make sure you are fully aware of the impact of this operation when you perform it on a live network.

Procedure

1.     Enter system view.

system-view

2.     Set the mode of RoCEv2 traffic analysis.

netanalysis rocev2 mode { bidir | single }

By default, RoCEv2 traffic analysis is disabled, and the mode of RoCEv2 traffic analysis is not set.

Enabling RoCEv2 traffic statistics collection

About this task

Perform this task to enable NetAnalysis to analyze RoCEv2 traffic and send the analysis results to the NAP. You can use an ACL to match the RoCEv2 traffic of interest. The deny or permit action in the ACL does not take effect. NetAnalysis supports the following rules of advanced ACLs:

·     Rule 1—Matches the UDP protocol and destination IPv4 address.

·     Rule 2—Matches the UDP protocol and source IPv4 address.

·     Rule 3—Matches the UDP protocol and source and destination IPv4 addresses.

To ensure correct collection and reporting of RoCEv2 traffic statistics, use the rules supported by NetAnalysis. For more information about ACLs, see ACL and QoS Configuration Guide.

Procedure

1.     Enter system view.

system-view

2.     Enable RoCEv2 traffic statistics collection.

netanalysis rocev2 statistic { global | acl name acl-name }

By default, RoCEv2 traffic statistics collection is disabled.

Enabling RoCEv2 packet loss analysis globally

About this task

This task enables the device to perform RoCEv2 packet loss analysis for all received RoCEv2 packets.

Restrictions and guidelines

For RoCEv2 packet loss analysis to take effect, first enable bidirectional mode for RoCEv2 traffic analysis.

Procedure

1.     Enter system view.

system-view

2.     Enable RoCEv2 packet loss analysis globally.

netanalysis rocev2 drop global

By default, RoCEv2 packet loss analysis is disabled globally.

Enabling AI ECN for RoCEv2 traffic statistics collection

About this task

AI ECN allows the device to collect RoCEv2 traffic statistics on a per-session basis on the outgoing interfaces for RoCEv2 traffic and send the RoCEv2 traffic statistics to the NDA. Based on the RoCEv2 traffic statistics, the NDA automatically adjusts the ECN threshold for lossless queues to ensure low latency and high throughput for lossless traffic. For more information about ECN, see QoS configuration in ACL and QoS Configuration Guide.

Procedure

1.     Enter system view.

system-view

2.     Enable AI ECN for RoCEv2 traffic statistics collection.

netanalysis rocev2 ai-ecn enable

By default, AI ECN is disabled for RoCEv2 traffic statistics collection.

Setting the interval for reporting RoCEv2 traffic statistics to the NDA

About this task

Perform this task to adjust the interval at which the device reports RoCEv2 traffic statistics to the NDA.

Procedure

1.     Enter system view.

system-view

2.     Set the interval for reporting RoCEv2 traffic statistics to the NDA.

netanalysis rocev2 report-interval interval

By default, the device reports RoCEv2 traffic statistics to the NDA at an interval of 10 seconds.

Setting the aging timer for inactive RoCEv2 flows

About this task

When an inactive RoCEv2 flow ages out, the device outputs the related traffic statistics to the NDA, deletes these traffic statistics from the RoCEv2 cache, and deletes the related flow entries.

Procedure

1.     Enter system view.

system-view

2.     Set the aging timer for inactive RoCEv2 flows.

netanalysis rocev2 timeout inactive seconds

By default, the aging timer for inactive RoCEv2 flows is set to 30 seconds.

Display and maintenance commands for NetAnalysis for RoCEv2 traffic

Execute display commands in any view and reset commands in user view.

 

Task	Command
Display configuration and status of the RoCEv2 cache.	display netanalysis rocev2 cache [ source source-ip | destination destination-ip ]*
Display RoCEv2 traffic statistics.	display netanalysis rocev2 statistics
Clear RoCEv2 traffic statistics.	reset netanalysis rocev2 statistics

 

NetAnalysis for RoCEv2 traffic configuration examples

Example: Configuring NetAnalysis to analyze RoCEv2 traffic

Network configuration

As shown in Figure 6, configure NetAnalysis on the device to analyze the bidirectional RoCEv2 traffic sent between the server and the client and output the RoCEv2 traffic statistics to the NDA.

Figure 6 Network diagram

‌

Procedure

1.     Assign an IP address to each interface, as shown in Figure 6. (Details not shown.)

2.     Configure NetAnalysis to analyze RoCEv2 traffic:

# Enable bidirectional RoCEv2 traffic analysis.

<Device> system-view

[Device] netanalysis rocev2 mode bidir

# Enable global RoCEv2 traffic statistics collection.

[Device] netanalysis rocev2 statistics global

Verifying the configuration

# Display configuration and status of the RoCEv2 cache after the device has been operating for a period of time.

[Device] display netanalysis rocev2 cache

NOTE:

S2D: source to destination              D2S: destination to source

RTT: round trip time                    RPT: packet throughput in read mode

WPT: packet throughput in write mode    SPT: packet throughput in send mode

I: input          O: output             L: local          R: remote

NetAnalysis cache information:

-----------------------------------------------------------------------------

Flow created at        Service type

Src IP                 Src QP        S2D RTT      S2D RPT      S2D SPT/WPT

S2D NAK Pkts           S2D Interface(I)           S2D Interface(O)

Dst IP                 Dst QP        D2S RTT      D2S RPT      D2S SPT/WPT

D2S NAK Pkts           D2S Interface(I)           D2S Interface(O)

-----------------------------------------------------------------------------

01/22/2019 09:08:15    RC

11.110.2.2             93309         50           11           11

2                      WGE1/0/1(L)                WGE1/0/2(L)

12.110.2.2             85353         50           11           11

8373                   WGE1/0/2(L)                WGE1/0/1(L)

# Display RoCEv2 traffic statistics.

[Device] display netanalysis rocev2 statistics

Last statistics resetting time: Never

--------------------------------------------------------------------------------

Received packets: 1833088

--------------------------------------------------------------------------------

Type

         Active         Aged           Created         Reported

        (Sessions)     (Sessions)     (Sessions)      (Sessions)

--------------------------------------------------------------------------------

RoCEv2

        1               0               1              10

--------------------------------------------------------------------------------