Contents

Configuring RDMA telemetry· 1

About RDMA telemetry· 1

System model 1

Measurement indexes· 2

Read and write I/O interaction processes· 2

Restrictions and guidelines· 5

Configuring RDMA telemetry· 6

Display and maintenance commands for RDMA telemetry· 7

RDMA telemetry configuration examples· 7

Example: Configuring RDMA telemetry· 7

 

 

Configuring RDMA telemetry

About RDMA telemetry

RDMA telemetry is an intelligent network monitoring technology that divides the data center storage network into three segments. It measures the data transmission latency (I/O quality) of read and write operations for each segment in real time, and generates visual network quality reports to help administrators to quickly locate performance bottlenecks.

System model

The RDMA telemetry measurement system contains the following components:

·     Controller and analyzer.

¡     The controller deploys measurement configuration to devices through NETCONF interfaces.

¡     Devices report measurement data to the analyzer through gRPC. Then, the analyzer displays I/O performance indexes (latency).

·     Measurement devices: Network devices that support RDMA telemetry.

¡     Compute-side port—Port that connects a device to the compute node (such as a Web server or mail server). It identifies NVMe and RDMA packets, measures compute-side latency, and reports the measurement data to the analyzer.

¡     Storage-side port—Port that connects a device to the storage array. It identifies NVMe and RDMA packets, measures storage-side latency, and reports the measurement data to the analyzer.

RDMA telemetry divides the data center storage network into the following segments:

·     Compute side—The segment between a network device and the compute node.

·     Storage side—The segment between a network device and the storage array.

·     Network path—The transmission network between the compute node and the storage array.

Figure 1 RDMA telemetry system model

 

Measurement indexes

Based on the I/O interaction process, RDMA telemetry periodically performs measurement and calculates the average value for each of the following objects:

·     Data Preparation Latency (DPL)—Applicable only to write I/O operations. It is used for evaluating the data preparation efficiency on the compute side.

If the DPL is high, a CPU or memory bottleneck (such as thread blocking) might exist on the compute side.

·     Round-Trip Time (RTT)—Round-trip latency of packet transmission in the network used for measuring network performance. It is calculated by subtracting the total storage-side latency (IOL2) from the total compute-side latency (IOL1).

If the RTT suddenly increases, network congestion, BGP route flapping, or NIC queue backlog might exist.

·     Data Access Latency (DAL)—The time the storage device takes to process I/O requests. Read DAL and write DAL are measured separately, and they reflect different processing stages on the storage side.

If the DAL is high, high SSD latency, RAID controller bottleneck, or storage software stack overloading might exist.

Figure 2 Measurement indexes for RDMA telemetry

 

Read and write I/O interaction processes

Key packets and measurement indexes

Figure 3 shows the key packets that pass through the compute-side and storage-side ports, and the measurement indexes recorded by the devices in the read and write I/O interaction processes.

Figure 3 Key packets and measurement indexes in the read and write I/O interaction processes

 

Table 1 shows the key packets and the measurement indexes recorded by devices when the compute side reads data from the storage side.

Table 1 Key packets and measurement indexes in the read I/O interaction process

Step	Direction	Packet type	RDMA operation	Data recorded by devices
1	From compute side to storage side	NVMe CMND (read request)	RDMA Send Only	·     The compute-side port records the timestamp when packet 1 was sent and starts timing for IOL1. ·     The storage-side port records the timestamp when packet 1 was received (timestamp when the storage side received the request) to calculate the DAL and starts timing for IOL2.
2	From storage side to compute side	NVMe_DATA (first data packet)	RDMA Write First	The storage-side port records the timestamp when packet 2 was sent to calculate the read DAL.
3	From storage side to compute side	NVMe_DATA (final data packet)	RDMA Write Last	No data recorded.
4	From storage side to compute side	NVMe RSP (response completion)	RDMA Send Only Inval.	·     The storage-side port records the timestamp when packet 4 was sent and ends timing for IOL2. ·     The compute-side port records the timestamp when packet 4 was received and ends timing for IOL1.

 

Table 2 shows the key packets and the measurement indexes recorded by devices when the compute side writes data to the storage side.

Table 2 Key packets and measurement indexes in the write I/O interaction process

Step	Direction	Packet type	RDMA operation	Telemetry measurement data
5	From compute side to storage side	NVMe CMND (write request)	RDMA Send Only	·     The compute-side port records the timestamp when packet 5 was sent and starts timing for IOL1. ·     The storage-side port records the timestamp when packet 5 was received to calculate the write DAL.
6	From storage side to compute side	NVMe Read Request (data pull request)	RDMA Read Request	·     The storage-side port records the timestamp when packet 6 was sent to calculate the DAL. ·     The compute-side port records the timestamp when packet 6 was received to calculate the DPL.
7	From compute side to storage side	NVMe_DATA (first data packet)	RDMA Read Resp First	The compute-side port records the timestamp when packet 7 was sent to calculate the DPL.
8	From compute side to storage side	NVMe_DATA (final data packet)	RDMA Read Resp Last	No data recorded.
9	From storage side to compute side	NVMe RS (write acknowledgement)	RDMA Send Only Inval.	·     The storage-side port records the timestamp when packet 9 was sent and ends timing for IOL2. ·     The compute-side port records the timestamp when packet 9 was received and ends timing for IOL1.

 

Calculating measurement indexes

·     Read DAL: time difference between receiving a read request (packet 1) and sending the first data packet (packet 2) on the storage-side port.

Read DAL = timestamp when packet 2 was sent − timestamp when packet 1 was received

·     Write DAL: time difference between receiving a write request (packet 5) and sending a data pull request (packet 6) on the storage-side port.

Write DAL = timestamp when packet 6 was sent − timestamp when packet 5 was received

·     DPL: time difference between receiving the data pull quest (packet 6) and sending the first data response (packet 7) on the compute side.

DPL = timestamp when packet 7 was sent − timestamp when packet 6 was received

·     IOL1 (total compute-side latency):

IOL1 = response timestamp − request timestamp = timestamp when packet 4 was received − timestamp when packet 1 was sent

¡     Read operation: time difference between sending packet 1 and receiving packet 4 on the compute side.

¡     Write operation: time difference between sending packet 5 and receiving packet 9 on the compute side.

·     IOL2 (total storage-side latency):

IOL2 = timestamp for storage-side response sending − timestamp for storage-side request receiving = timestamp when packet 4 was sent − timestamp when packet 1 was received

¡     Read operation: time difference between receiving packet 1 and sending packet 4 on the storage side.

¡     Write operation: time difference between receiving packet 5 and sending packet 9 on the storage side.

·     RTT = IOL1 − IOL2

Data acquisition process

The following uses data acquisition of a common user as an example to describe the complete RDMA telemetry measurement process.

1.     The user sends a request, and the compute side processes the request.

The user clicks an app, which triggers the compute side (such as a Web server) to generate a data request. The application service encapsulates the request into an NVMe over Fabrics (NoF) packet (such as RDMA Send Only).

2.     The request traverses network devices (RTT measurement points).

The compute side sends the request through the RDMA NIC, and the network devices forward the request to the storage side. The network devices record packet timestamps and associate requests and responses (such as packet 1 and packet 4).

3.     The storage-side (DAL measurement point) processes the request.

The storage-side port receives the request (such as packet 1), reads data from flash memory, and then sends data to the compute side (such as packet 2).

The storage device directly writes or pulls data through RDMA write or read, respectively.

The DAL is the time difference between receiving the request (packet 1) and sending data (packet 2) on the storage side.

4.     Data is returned to the user (end-to-end closed loop).

The storage-side data is returned to the compute side. The compute-side application service assembles the response data and sends it back to the user in an HTTP message.

Restrictions and guidelines

RDMA telemetry is supported only in expert operating mode. For more information about the device operating modes, see device management configuration in Fundamentals Configuration Guide.

You can configure RDMA telemetry only on Ethernet interfaces, and you cannot configure it on member ports of an aggregation group.

RDMA telemetry does not support measuring VPN or M-LAG traffic. For more information about VPN, see MPLS Configuration Guide. For more information about M-LAG configuration, see Layer 2—LAN Switching Configuration Guide.

Only Layer 3 known unicast transmission supports RDMA telemetry measurement.

RDMA telemetry does not support measuring traffic in VXLAN networks. For more information about VXLAN, see VXLAN Configuration Guide.

RDMA telemetry periodically performs sampling and measurement and calculates average values. Because it does not sample each I/O, the latency statistics about non-sampled packets cannot be displayed.

RDMA telemetry does not perform full measurement on all I/Os. Some latency statistics are not effectively monitored.

In the current software version, RDMA telemetry does not measure small-byte I/Os.

When the compute-side and storage-side ports reside on the same card, the packet receiver and sender cannot be distinguished. Inbound packets might be sent to the analyzer in advance, distorting latency measurement.

When you configure RDMA telemetry, follow these guidelines:

·     The RDMA telemetry feature is mutually exclusive with the netanalysis udp command.

·     For RDMA telemetry to operate correctly, do not use it together with the ip load-sharing mode per-packet or ip load-sharing mode per-flow ingress-port command.

·     As a best practice, do not use IPv6 flow filtering rules for RDMA telemetry together with the unified flow analysis (UFA) feature.

For more information about the netanalysis udp command and UFA, see NetAnalysis configuration in Network Management and Monitoring Configuration Guide. For more information about the ip load-sharing mode command, see IP forwarding basics configuration in Layer 3—IP services Configuration Guide.

Configuring RDMA telemetry

Table 1 Enter system view.

system-view

2.     Enter RDMA-telemetry view.

rdma-telemetry

3.     Configure a flow filtering rule for RDMA telemetry.

flow flow-name [ ipv6 ] { destination destination-ip-address | source source-ip-address } *

By default, no flow filtering rule is configured for RDMA telemetry.

RDMA telemetry measures only the service traffic that matches filtering rules.

4.     Set the measurement interval for I/O quality visualization.

interval interval-value

By default, the measurement interval is 10 seconds for I/O quality visualization.

After you configure RDMA telemetry, the device calculates the DAL, DPL, and IOL for each read and write operation. It also calculates the average DAL, average DPL, and average IOL within a measurement interval. To view the average statistics, execute the display rdma-telemetry statistics command or configure the device to report the statistics to the analyzer through gRPC.

5.     Enable global I/O quality visualization.

rdma-telemetry global enable

By default, global I/O quality visualization is disabled.

6.     Enter interface view.

interface interface-type interface-number

7.     Enable I/O quality visualization on the interface.

rdma-telemetry enable

By default, I/O quality visualization is disabled on an interface.

Display and maintenance commands for RDMA telemetry

Execute display commands in any view.

 

Task	Syntax
Display RDMA telemetry statistics.	display rdma-telemetry statistics

 

RDMA telemetry configuration examples

Example: Configuring RDMA telemetry

Network configuration

A PC accesses the Web server to read and write webpages. The process is as follows:

Table 2 The PC (Web access endpoint) sends read and write requests to the compute node (Web server).

2.     The compute node reads data from and writes data to the storage array.

3.     The compute node returns the result to the PC.

Configure RDMA telemetry to measure and display the latency of each segment in a single I/O read/write operation. The latency statistics include the compute-side latency, the transmission network latency, and the storage-side latency.

Figure 4 Network diagram

 

Table 3 Interface label and interface name mappings

Interface label	Interface name
Interface1	Ten-GigabitEthernet 1/0/1

 

Prerequisites

Make sure the compute node, devices, and storage array can reach each other, and the devices and the analyzer can reach each other. (In this example, RDMA telemetry is directly configured on the devices. You do not need to deploy a controller to issue configuration to the devices.)

Configuring Device A

Table 1 Configure gRPC to report RDMA telemetry measurement data to the analyzer:

# Enable the gRPC service.

<Sysname> system-view

[Sysname] sysname DeviceA

[DeviceA] grpc enable

# Create a sensor group named test, and add sensor path rdmatelemetry/rdmaserviceevent.

[DeviceA] telemetry

[DeviceA-telemetry] sensor-group test

[DeviceA-telemetry-sensor-group-test] sensor path rdmatelemetry/rdmaserviceevent

[DeviceA-telemetry-sensor-group-test] quit

# Create a destination group named collector1 and add a collector to the group.

[DeviceA-telemetry] destination-group collector1

[DeviceA-telemetry-destination-group-collector1] ipv4-address 10.1.3.1 port 50050

[DeviceA-telemetry-destination-group-collector1] quit

# Configure a subscription named A to bind sensor group test with destination group collector1.

[DeviceA-telemetry] subscription A

[DeviceA-telemetry-subscription-A] sensor-group test

[DeviceA-telemetry-subscription-A] destination-group collector1

[DeviceA-telemetry-subscription-A] quit

[DeviceA-telemetry] quit

2.     Configure I/O quality visualization:

# Enable I/O quality visualization on Ten-GigabitEthernet 1/0/1.

[DeviceA] interface ten-gigabitethernet 1/0/1

[DeviceA-Ten-GigabitEthernet1/0/1] rdma-telemetry enable

[DeviceA-Ten-GigabitEthernet1/0/1] quit

# Enable global I/O quality visualization.

[DeviceA] rdma-telemetry

[DeviceA-rdma-telemetry] rdma-telemetry global enable

# Configure a flow filtering rule for RDMA telemetry.

[DeviceA-rdma-telemetry] flow test source 10.1.1.1 destination 10.1.2.1

Configuring Device B

Table 1 Configure gRPC to report RDMA telemetry measurement data to the analyzer:

# Enable the gRPC service.

<Sysname> system-view

[Sysname] sysname DeviceB

[DeviceB] grpc enable

# Create a sensor group named test, and add sensor path rdmatelemetry/rdmaserviceevent.

[DeviceB] telemetry

[DeviceB-telemetry] sensor-group test

[DeviceB-telemetry-sensor-group-test] sensor path rdmatelemetry/rdmaserviceevent

[DeviceB-telemetry-sensor-group-test] quit

# Create a destination group named collector1 and add a collector to the group.

[DeviceB-telemetry] destination-group collector1

[DeviceB-telemetry-destination-group-collector1] ipv4-address 10.1.3.1 port 50050

[DeviceB-telemetry-destination-group-collector1] quit

# Configure a subscription named A to bind sensor group test with destination group collector1.

[DeviceB-telemetry] subscription A

[DeviceB-telemetry-subscription-A] sensor-group test

[DeviceB-telemetry-subscription-A] destination-group collector1

[DeviceB-telemetry-subscription-A] quit

[DeviceB-telemetry] quit

2.     Configure I/O quality visualization:

# Enable I/O quality visualization on Ten-GigabitEthernet 1/0/1.

[DeviceB] interface ten-gigabitethernet 1/0/1

[DeviceB-Ten-GigabitEthernet1/0/1] rdma-telemetry enable

[DeviceB-Ten-GigabitEthernet1/0/1] quit

# Enable global I/O quality visualization.

[DeviceB] rdma-telemetry

[DeviceB-rdma-telemetry] rdma-telemetry global enable

# Configure a flow filtering rule for RDMA telemetry.

[DeviceB-rdma-telemetry] flow test source 10.1.1.1 destination 10.1.2.1

Verifying the configuration

Table 1 Display RDMA telemetry statistics on Device A. Verify that you can view the latency statistics about read and write operations (AIOL, ADAL, and ADPL).

<DeviceA> display rdma-telemetry statistics

AIOL: Average I/O latency(us)

ADAL: Average data access latency(us)

ADPL: Average data preparation latency(us)

R: Read   W: Write

Measurement interval: 10s

Measured at: 2025-8-20 08:20:19

-----------------------------------------------------------------------------------

SIP                                 DIP                                   Type

AIOL(R/W)             ADAL(R/W)             ADPL(R/W)            

-----------------------------------------------------------------------------------

10.1.1.1                             10.1.2.1                              Initiator

1345/1246             --/--                 --/48                

2.     Display RDMA telemetry statistics on Device B. Verify that you can view the latency statistics about read and write operations (AIOL, ADAL, and ADPL).

<DeviceB> display rdma-telemetry statistics

AIOL: Average I/O latency(us)

ADAL: Average data access latency(us)

ADPL: Average data preparation latency(us)

R: Read   W: Write

Measurement interval: 10s

Measured at: 2025-8-20 08:20:19

-----------------------------------------------------------------------------------

SIP                                  DIP                                  Type

AIOL(R/W)             ADAL(R/W)             ADPL(R/W)            

-----------------------------------------------------------------------------------

10.1.1.1                             10.1.2.1                              Target

1332/1224             34/56                 --/--                

Configuration files

·     Device A:

#

sysname DeviceA

#

grpc enable

#

telemetry

  sensor-group test

    sensor path rdmatelemetry/rdmaserviceevent

  destination-group collector1

    ipv4-address 192.168.134.1 port 50050

  subscription A

    sensor-group test

    destination-group collector1

#

interface Ten-GigabitEthernet1/0/1

  rdma-telemetry enable

#

rdma-telemetry

  rdma-telemetry global enable

  flow test source 10.1.1.1 destination 10.1.2.1

#

·     Device B:

#

sysname DeviceB

#

grpc enable

#

telemetry

  sensor-group test

    sensor path rdmatelemetry/rdmaserviceevent

  destination-group collector1

    ipv4-address 192.168.134.1 port 50050

  subscription A

    sensor-group test

    destination-group collector1

#

interface Ten-GigabitEthernet1/0/1

  rdma-telemetry enable

#

rdma-telemetry

  rdma-telemetry global enable

  flow test source 10.1.1.1 destination 10.1.2.1

#