· The access bandwidth upgrades from 10 Gbps to 25 Gbps or 100 Gbps. The cache space growth cannot match the access bandwidth growth, which proposes higher requirements for the network forwarding capacity and service guarantee.

· With the increase of Remote Direct Memory Access (RDMA) applications that require lossless Ethernet deployment, more precise cache monitoring and traffic monitoring are needed.

Traditional network monitoring technologies cannot properly face new challenges because of the following shortcomings:

· The monitoring device sends poll packets outside the service network to query the network device status. In this way, the monitoring device cannot obtain device operating statistics in real time and the poll packets occupy network bandwidth.

· The forwarding path and forwarding delay information cannot be visualized.

The telemetry technology provides a solution for better implementation of network O&M visualization. It includes techniques such as Google Remote Procedure Call (gRPC), In-band Network Telemetry (INT), telemetry stream, Mirror On Drop (MOD), Transient Capture Buffer (TCB).

· gRPC is an open source remote procedure call (RPC) system initially developed at Google. It uses HTTP 2.0 and provides network device configuration and management methods that support multiple programming languages. The server and client developers can develop server-side and client-side custom capabilities in the gRPC framework. Integrated with the gRPC application, the device serializes the operating information such as cache, memory, CPU usage, and logs through the protocol buffer code format and proactively pushes the operating information to the collector.

· INT is a network monitoring technique designed to collect and report device operating status at the data forwarding plane. INT allows you to obtain the following information on the forwarding path:

¡ Device information.

¡ Ingress port, egress port, and queue information of packets on each device.

¡ Timestamp information.

· Similar to INT, telemetry stream can collect the ingress port, egress port, and timestamp information of packets. The difference is that you can configure telemetry stream only on devices that require data collection instead of the whole network. Telemetry stream can calculate the delay generated during packet forwarding on a certain path by using the timestamps reported by nodes.

· MOD monitors the packet drop events during the internal forwarding process on a device. Once MOD detects a dropped packet within the device, it immediately collects the packet drop time, packet drop reason, and characteristics of the dropped packet and sends the collected information to the collector. Then, the network administrator can learn the packet drop event on the device.

· TCB monitors packet drop events of Memory Management Unit (MMU) queues. After you enable TCB, the system will continuously monitor the MMU queues. When a packet is dropped in a queue, the system collects the packet drop time, packet drop reason, and original data of the dropped packet, and reports the collected information to the NMS through gRPC. Then, the network administrator can learn the packet drop event on the device.

Benefits

gRPC

H3C gRPC-based telemetry (gRPC in dial-out mode as shown in "About gRPC") uses the "push" mode to push the device operating information to collectors (one subscription for continuous returned data flows). This avoids the impact of polling on the device and network traffic.

Figure 1 Comparison between the pull and push modes

gRPC-based telemetry also features fast response speed, high real-time performance, high precision, comprehensive data collection, and structured data.

Table 1 Comparison between network management protocols

Network management protocol	Operating mode	Data scope	Data granularity	Structured data
SNMP get	Pull mode (polling, low performance, limited scale)	All	Minutes	Structured
SNMP trap	Pull mode	Alarms	Seconds	Structured
Syslog	Pull mode	Events	Seconds	Unstructured
gRPC in dial-out mode	Pull mode	All	Seconds	Structured

INT

INT visualizes forwarding paths and forwarding delay information.

INT provides visibility into the following information on the forwarding path:

· Ingress port, egress port, and queue information of each device.

· Ingress timestamp and egress timestamp.

· Queue congestion information.

On the last hop in the forwarding path, INT encapsulates monitored packets with the UDP header and IP header and sends the packets to the collector. Then, the NMS software deployed on the collector analyzes the monitored data and extracts useful information.

INT provides the following benefits:

· INT is supported by hardware. It collects and reports information at the data forwarding plane without the participation of the control plane. This enables INT to reflect the real packet forwarding situation without increasing the CPU load of network devices.

· One configuration deployment for continuous data reporting.

· You can configure the sampling rate of original packets to reduce bandwidth occupancy.

· You can configure QoS policies to flexibly match original packet that require path detection.

Telemetry stream

Telemetry stream visualizes forwarding delay information.

Telemetry stream helps us monitor the ingress port, egress port, and timestamp information for packets. The collector can calculate the delay generated when a packet is forwarded by a node based on the timestamp reported by the node. In addition, the collector can calculate the delay generated when a packet is forwarded on a certain path by using the timestamps reported by multiple nodes.

Telemetry stream provides the following benefits:

· Telemetry stream supports hardware-based line-speed transmission.

· One configuration deployment for continuous data reporting.

· You can configure the sampling rate of original packets to reduce bandwidth occupancy.

· You can configure ACLs to flexibly match original packets that require information collection.

MOD

You can configure flow groups and ACLs to flexible match traffic to be monitored. You can also configure multiple packet drop reasons and configure MOD to send packets through UDP or gRPC to the collector as required.

TCB

TCB provides the following benefits:

· Proactive exception reporting.

Once a packet is dropped in an MMU queue, the packet drop information is proactively sent to an NMS.

· Efficient data transmission.

TCB uploads data through gRPC.

· Chip-level O&M.

TCB monitors packet drop events of MMU queues in real time inside the network device chips.

· Reliable service operation.

TCB can detect packet drop events in real time, which minimizes the impact on services.

· Deep control over the interior of network devices.

TCB continuously monitors the MMU queues inside the device forwarding chips, which is one of the important methods for network visualization.

About gRPC

gRPC protocol stack layers

Table 2 describes the gRPC protocol stack layers.

Table 2 gRPC protocol stack layers

Layer	Description
Content layer	Defines the data of the service module. Two parties of communication must notify each other of the data models that they are using.
Protocol buffer encoding layer	gRPC encodes data by using the protocol buffer code format.
gRPC layer	Defines the protocol interaction format for remote procedure calls.
HTTP 2.0 layer	Carries gRPC.
TCP layer	TCP provides connection-oriented, reliable, and sequential data links.

gRPC network architecture

As shown in Figure 2, a gRPC network uses the client/server model and transports messages between the client and the server over HTTP 2.0.

Figure 2 gRPC network architecture

When the device acts as the server, the gRPC mechanism is as follows:

1. The gRPC server listens to connection requests from clients at the gRPC service port.

2. A user runs the gRPC client application to log in to the gRPC server, and uses methods provided in the .proto file to send requests.

3. The gRPC server responds to requests from the gRPC client.

H3C devices can act as gRPC servers or clients.

Telemetry technology based on gRPC

As shown in Figure 3, after a gRPC connection is established between the device and NMSs, the NMSs can subscribe to data of modules on the device.

Figure 3 Telemetry technology based on gRPC

Telemetry modes

The device supports the following telemetry modes:

· Dial-in mode—The device acts as a gRPC server and the collectors act as gRPC clients. A collector initiates a gRPC connection to the device to subscribe to device data.

Dial-in mode typically applies to small networks and scenarios where collectors deploy configurations to devices.

· Dial-out mode—The device acts as a gRPC client and the collectors act as gRPC servers. The device initiates gRPC connections to the collectors and pushes device data to the collectors as configured.

Dial-out mode typically applies to large networks where the collectors must collect data from a large number of devices.

Protocol buffer code format

Google Protocol Buffers provide a flexible mechanism for serializing structured data. Different from XML code and JSON code, the protocol buffer code is binary and provides higher performance. However, JSON code is used for service data.

Proto definition files

You can define data structures in a proto definition file. Then, you can compile the file with utility protoc to generate code in a programing language such as Java and C++. Using the generated code, you can develop an application for a collector to communicate with the device.

H3C provides proto definition files in both dial-in mode and dial-out mode.

Proto definition files in dial-in mode

· Public proto definition files.

The grpc_service.proto file defines the public RPC methods in dial-in mode (for example, login and logout).

· Proto definition files for service modules.

The dial-in mode supports proto definition files for the following service modules: Device, Ifmgr, IPFW, LLDP, and Syslog.

Proto definition file in dial-out mode

The grpc_dialout.proto file defines the public RPC methods in dial-out mode.

NOTE:

The service module data in dial-out mode on H3C devices is defined in XML format, which is the same as the NETCONF XML API data format. For information about how to obtain and use the service module data, see "gRPC usage guidelines."

Obtaining proto definition files

To obtain proto files, contact H3C Support.

About INT

INT networking model

An INT network contains the following INT-enabled devices: one entry node, several transit nodes, and one exit node.

Figure 4 INT network

As shown in Figure 4, the INT network contains the following components:

· Entry node—The first device on the INT network samples packets and mirrors packets to the INT processor within the device, adds the INT header and monitoring information to the packets, and sends packets to a transit node.

· Transit node—A transit node recognizes INT packets, adds monitoring information to the packets, and sends packets to a downstream node.

· Exit node—The last hop device on the INT network automatically recognizes INT packets, adds the monitoring information to the packets, encapsulates the monitoring information with the UDP header and IP header according to the configured encapsulation parameters, and sends the packets to the collector.

· Ingress port—For the entry node, this port is the input interface of original packets. For the transit nodes and exit node, this port is the input interface of INT packets.

· Egress port—For the entry node and transit nodes, this port is the output interface of INT packets. For the exit node, this port is the output interface of encapsulated packets.

· INT processor—The INT processor is a dedicated processor in the CPU used for processing INT packets.

Metadata that INT can collect

INT can collect and monitor the following metadata:

· Device ID—The device ID of each device on the packet forwarding path, which is the device ID specified when you configure INT.

· Ingress port ID—Logical input interface of packets on each node in the INT network.

· Ingress timestamp—The local time on the device when a packet enters the ingress port. For the entry node, it is the time when an INT packet enters the loopback interface.

· Egress port ID—Logical output interface of packets on each node in the INT network.

· Egress timestamp—The local time on the device when a packet leaves the egress port.

· Cache information—Options include:

¡ Queue ID—ID of the queue that caches original packets.

¡ ECN information.

INT packet types

INT has two packet types: INT over TCP and INT over UDP.

An original TCP packet is called an INT-over-TCP packet after it is mirrored and inserted with an INT header.

An original UDP packet is called an INT-over-UDP packet after it is mirrored and inserted with an INT header.

Figure 5 INT over TCP

Figure 6 INT over UDP

Description of related fields:

· INT Probe HDR—Header inherent to INT.

· MD #1-N—Inserted metadata.

How INT works

INT supports common INT and flexible INT. The two types of INT have the following differences:

· Common INT—Each node needs to be configured with an INT role on its input interface: ingress, transit, and egress. Traffic flows are defined on the entry node by using a QoS policy. INT flows are automatically identified on the transit node and exit node and processed according to configured actions. On each input interface in the path, you can perform INT processing only on the flows defined on the entry node.

· Flexible INT—No device role needs to be configured on each node. On each node, an ACL can be used to define a flow and an action used to take on the defined flow. For the same flow, the original packets are matched on the entry node, and the INT packets are matched on the transit node and exit node. You can define multiple flows on an interface and take different actions on different flows.

Common INT is easy to configure and is recommended. Use flexible INT only when you need to process multiple flows on an interface.

Common INT

As shown in Figure 7, the nodes in common INT perform the following functions:

· Entry node:

On the entry node, the ingress port uses a QoS policy to sample matching packets, and mirrors sampled packets to the INT processor. Then, the INT processor adds an INT header to the INT packet and loops it back to the ingress port. The ingress port identifies the looped-back INT packet according to the INT mark, adds collected data to it, and forwards it to the egress port. The egress port adds collected data to the INT packet and sends it to the transit node.

· Transit node:

On the transit node, the ingress port identifies the INT packet according to the INT mark, adds collected data to it, and forwards it to the egress port. The egress port adds collected data to the INT packet, and sends it to the exit node.

· Exit node:

On the exit node, the ingress port identifies the INT packet according to the INT mark, adds collected data to the INT packet, encapsulates the INT packet with the configured parameters, and sends it to the collector.

Figure 7 Common INT

Flexible INT

As shown in Figure 7, the nodes in flexible INT perform the following functions:

· Entry node:

On the entry node, the ingress port uses an ACL to sample matching packets, and mirrors sampled packets to the INT processor. Then, the INT processor adds an INT header to the INT packet and loops it back to the ingress port. The ingress port identifies the looped-back INT packet according to an ACL and adds collected data to it, and forwards it. The egress port adds collected data to the INT packet and sends it to the transit node.

· Transit node:

On the transit node, the ingress port uses an ACL to identify INT packets, adds collected data to INT packets, and forwards them to the egress port. The egress port adds collected data to INT packets and sends them to the exit node.

· Exit node:

On the exit node, the ingress port uses an ACL to identify INT packets and mirrors them to the INT processor. The INT processor encapsulates the INT packets and sends them to the collector.

About telemetry stream

As shown in Figure 8, telemetry stream works as follows:

1. Telemetry stream samples a matching packet on the input interface and copies it to the telemetry stream processor in the device.

2. The telemetry stream processor loops the packet back to the input interface.

3. The input interface identifies the looped-back packet, adds collected information, and forwards it to the output interface according to the routing table.

4. The output interface identifies the looped-back packet, adds collected information, and sends it to the telemetry stream processor.

5. The telemetry stream processor encapsulates the packet and routes it to the collector.

The collector collects data of all devices that a specific flow passes through according to the data reported by multiple devices. Although the mirrored packet and the original packet do not enter or leave the device at the same time, the forwarding processes for the packets are the same. The delay calculated based on the carried timestamp represents the delay of the original packet.

Figure 8 Telemetry stream working mechanism

About MOD

Traffic that MOD can monitor

MOD monitors packets that match flow entries.

Flow entries are generated based on the flow group feature on the device. When you configure the flow group feature, you must specify an ACL for packet filtering and specify a set of packet attributes used for generating flow entries (for example, the 5-tuple shown in the following table). When the device processes packets that match the specified ACL, the flow group extracts the specified attribute fields from the packet header. It takes packets with exactly the same attribute values as a flow and generates a flow entry for it.

Each flow entry has an aging time. When the number of packets matching the flow entry does not increase within the specified aging time, the flow entry will be deleted.

Destination IP address	Source IP addresses	Destination port number	Source port number	Protocol type	Aging time (minutes)	Packet count
10.0.0.20	10.0.0.2	3000	2000	IP	20	5120
10.0.0.20	10.0.0.2	3002	2000	IP	18	25
10.0.1.20	10.0.0.2	3000	2020	IP	15	813

A flow group can generate flow entries for MOD in the following modes, which have differences in device hardware resource occupancy.

· MOD mode—Occupies more hardware resources but has a lower burden on the CPU.

· Simple MOD mode—Has a higher burden on the CPU but saves hardware resources.

Packet drop reasons monitored by MOD

MOD can monitor various packet drop reasons. Table 3 displays the main supported packet drop reasons. You can specify concerned packet drop reasons based on the actual fault location requirements when you configure MOD.

Table 3 Packet drop reasons

Value	Description
higig-header-error	HiGig header error.
invalid-tpid	Invalid TPID.
ip-multicast-error	IP multicast packet error.
ipv4-dip-miss	The destination IPv4 address does not match a route entry. (This reason is also reported when the destination IPv4 address matches the default route.)
ipv4-l3-header-error	IPv4 header error.
parity-error	Parity error.
tunnel-header-error	Tunnel packet header error.
unknown-vlan	Unknown VLAN.

Information reporting method used by MOD

When the monitored traffic has dropped packets, and the packet drop reason is the one specified by the user, MOD collects the packet drop reason, packet drop time, and traffic characteristics. The collected information can be reported to the remote collector in the following methods, so that users can immediately understand the packet drop event on the device.

· The device encapsulates the collected information in a UDP packet, encapsulates the UDP packet based on the packet encapsulation information configured by the user, forwards the packet to the remote collector. (Default reporting method.)

· Configure gRPC on the device to send the collected information to the remote collector through gRPC.

About TCB

MMU overview

MMU is an important component of the switch chip. After an Ethernet frame enters the switch chip, it is processed in sequence by the ingress unit, MMU unit, and egress unit.

· The ingress unit is responsible for data packet check, address learning, VLAN processing, Layer 2 or Layer 3 forwarding, and inbound packet filtering.

· The MMU unit is responsible for data packet caching and scheduling.

· The egress unit is responsible for VLAN processing, CRC calculation, and outbound rate limiting of data packets.

TCB fundamentals

After you enable TCB for the MMU queue, TCB works as follows:

1. TCB samples the packets cached in the MMU queue and saves the sampled packets to the TCB ring buffer. At the same time, TCB records packet drop events in the TCB event buffer in real time.

2. When the MMU queue has a dropped packet, TCB can analyze the data in the ring buffer and event buffer locally to identify the characteristics of the dropped packet. TCB reports the packet drop reason and packet characteristics to the remote NMS through gRPC.

3. TCB can also avoid local data analysis. When packet drop occurs, TCB directly reports the packet drop reason and original data in the ring buffer to the remote NMS through gRPC. Not performing local analysis can save local device resources and reduce device load.

NOTE:

When the ring buffer is full, the packets already cached in the ring buffer will be overwritten by new packets.

TCB state machine

TCB state machine involves the following parameters:.

Parameter	Description
Start Threshold	Queue length above which packet capturing will be started.
Pre-sample Rate	Sampling rate when TCB is in the pre-trigger state.
Post-sample Rate	Sampling rate when TCB is in the post-trigger state.
Stop Threshold	Queue length below which packet capturing will be stopped.
Frozen Number	Number of captured packets.
Frozen Timer	Packet capture timer.

Figure 9 shows the TCB state machine.

1. The state machine enters the idle state when TCB is enabled.

2. When the queue length is greater than Start Threshold, the state machine transitions from the idle state to the pre-trigger state. The system captures packets by using the Pre-sample Rate.

3. When packet drops occur on the queue, the state machine transitions from the pre-trigger to the post-trigger state.

4. When the queue length is smaller than Stop Threshold, the state machine transitions from the post-trigger to the idle state. The system stops capturing packets.

5. When the number of captured packets reaches the Frozen Number or the capture timer expires, the state machine transitions from the post-trigger to the frozen state.

6. The state machine reports the packet drop reason and packet characteristics.

7. The state machine transitions from the frozen to pre-trigger state.

Figure 9 TCB state machine

Restrictions: Hardware compatibility with visualization features

Feature	S6805 S6825 S6850 S9850	S9820-64H	S9820-8C
gRPC	Supported	Supported	Supported
INT	Supported	Supported, transit node only	Supported, transit node only
MOD	Supported	Supported	Only simple MOD is supported
TCB	Supported	Supported	Not supported
Telemetry stream	Supported	Supported	Supported

NOTE:

For more information about the restrictions and guidelines on each product, see "Visualization deployment restrictions and guidelines".

Typical network model for visualization

The visualization technology can theoretically be applied to various networks. Data center networks typically require network O&M visualization. This document uses the typical spine-leaf data center network (applicable to scenarios with large amount of east-west traffic) as an example.

The spine-leaf data center network provides the following features:

· Network design layer:

¡ Leaf devices serve as server gateways. Layer 3 ECMP runs between leaf and spine devices, which enables high-speed data exchange between servers through multiple equal-cost paths and minimizes delay as much as possible.

¡ As a best practice, set the downlink and uplink bandwidth ratio to 2:1, 3:2, or 1:1 based on the actual data exchange requirements. For example, if you use an S6850-56HF switch as a leaf switch and connect the forty-eight 25G ports to servers and eight 100G ports to spine devices, the bandwidth convergence ratio is 3:2.

· Network O&M visualization layer:

¡ To realize real-time and device-initiated information collection, especially for cache usage and packet drops, configure gRPC on the related devices. The collected information includes the usage of the ingress buffer, egress buffer, and headroom buffer, ingress and egress buffer usage violation, and ingress and egress packet drop statistics.

¡ To control the packet forwarding paths (the specific equal-cost paths that are used) or the forwarding delay of each hop, configure INT or telemetry stream on the devices along the packet forwarding paths.

¡ To troubleshoot packet drop issues, you can configure MOD or TCB as required.

Figure 10 Data center spine-leaf network model

Visualization deployment restrictions and guidelines

gRPC usage guidelines

In dial-out mode, the device completes data collection through sensors. To specify a data source to be sampled, configure a sensor path. A sensor samples data in one of the following methods:

· Event-triggered—The device pushes data from the sensor path to collectors when certain events occur. For information about the event-triggered sensor paths, see NETCONF XML API Event Reference for the module.

· Periodic—The device pushes data from the sensor path to collectors at intervals. For information about periodic sensor paths, see NETCONF XML API Data Reference and GRPC API Reference for the module.

Only some modules have the GRPC API Reference. This document provides data tables dedicated to gRPC sampling. All or part of data tables do not support NETCONF. If a module has its GRPC API Reference, obtain the format definition of the data tables from the document first.

To view service modules that the device supports for data sampling in dial-out mode, execute the sensor path ? command in sensor group view. The following uses the S6850 switch series of the R6555P01 version as an example:

<System> system-view

[System] telemetry

[System-telemetry] sensor-group test

[System-telemetry-sensor-group-test] sensor path ?

acl/ ACL module

arp/ ARP module

bgp/ BGP module

buffermonitor/ BufferMonitor module

device/ Device module

diagnostic/ Diagnostic module

filesystem/ FileSystem module

fundamentals/ Fundamentals module

ipfw/ IPFW module

ifmgr/ Ifmgr module

lldp/ LLDP module

mac/ MAC module

mqc/ MQC module

route/ Route module

staticroute/ StaticRoute module

syslog/ Syslog module

ifmgr All events of the Ifmgr module

bgp_stream/ bgp_stream stream

bgp_stream All events of the bgp_stream module

diagnostic All events of the Diagnostic module

buffermonitor All events of the BufferMonitor module

lldp All events of the LLDP module

ipfw_event/ ipfw_event stream

ipfw_event All events of the ipfw_event module

syslog All events of the Syslog module

device All events of the Device module

To view data tables supported by a module, execute the sensor path module-name/? command. The following uses the buffermonitor module as an example.

[System-telemetry-sensor-group-path] sensor path buffermonitor/?

buffermonitor/boardconfigurations BoardConfigurations table

buffermonitor/boardegressdrops BoardEgressDrops table

buffermonitor/boardqueconfigurations BoardQueConfigurations table

buffermonitor/bufferusages BufferUsages table

buffermonitor/commbufferusages CommBufferUsages table

buffermonitor/commheadroomusages CommHeadroomUsages table

buffermonitor/commtotalbufferusages CommTotalBufferUsages table

buffermonitor/ecnandwredstatistics EcnAndWredStatistics table

buffermonitor/egressdrops EgressDrops table

buffermonitor/globalconfigurations GlobalConfigurations table

buffermonitor/headroomusages HeadroomUsages table

buffermonitor/ingressdrops IngressDrops table

buffermonitor/overruncounters OverRunCounters table

buffermonitor/pfcspeeds PFCSpeeds table

buffermonitor/pfcstatistics PFCStatistics table

buffermonitor/portqueconfigurations PortQueConfigurations table

buffermonitor/statistics Statistics table

buffermonitor/totalbufferusages TotalBufferUsages table

buffermonitor/boardoverrunevent BoardOverrunEvent event

buffermonitor/portqueoverrunevent PortQueOverrunEvent event

buffermonitor/portquedropevent PortQueDropEvent event

The boardoverrunevent, portqueoverrunevent, and portquedropevent tables use event-triggered sampling. For more information about the data format, see Comware BufferMonitor NETCONF XML API Event Reference. For the data format of other tables, see Comware V7 BufferMonitor GRPC API Reference.

NETCONF XML API files and GRPC API files are released with the software version release process and can be obtained from the software version documents.

INT usage guidelines

The S9820-64H and S9820-8C switches can only operate as transit nodes. The S6805, S6825, S6850, and S9850 switches can operate as entry nodes, transit nodes, and exit nodes.

When an S6805, S6825, or S9850 switch operates as the entry node, you must enable internal loopback on an interface. Enable internal loopback on a non-ingress interface in the interface group to which the ingress interface belongs. To view the interface group to which an interface belongs, execute the display qos-acl resource command. The interfaces in the Interfaces field belong to the same interface group. The INT internal loopback feature requires exclusive use of an interface. The interface cannot be used for other purposes.

Table 4 displays the mappings between the ingress or egress port IDs in the metadata of INT packet monitoring information and the ports on an S9820-64H switch.

Table 4 Mappings between the ingress or egress port IDs in the metadata of INT packet monitoring information and the ports on an S9820-64H switch

Panel port		Timestamp port ID	Ingress/egress port ID
Port before splitting	Port after splitting	Timestamp port ID	Ingress/egress port ID
H1/0/1	TW1/0/1:1	0	34
N/A	TW1/0/1:2	1	35
N/A	TW1/0/1:3	2	36
H1/0/2	TW1/0/1:4	3	37
H1/0/3	TW1/0/3:1	4	38
N/A	TW1/0/3:2	5	39
N/A	TW1/0/3:3	6	40
H1/0/4	TW1/0/3:4	7	41
H1/0/5	TW1/0/5:1	8	42
N/A	TW1/0/5:2	9	43
N/A	TW1/0/5:3	10	44
H1/0/6	TW1/0/5:4	11	45
H1/0/7	TW1/0/7:1	12	46
N/A	TW1/0/7:2	13	47
N/A	TW1/0/7:3	14	48
H1/0/8	TW1/0/7:4	15	49
H1/0/9	TW1/0/9:1	16	50
N/A	TW1/0/9:2	17	51
N/A	TW1/0/9:3	18	52
H1/0/10	TW1/0/9:4	19	53
H1/0/11	TW1/0/11:1	20	54
N/A	TW1/0/11:2	21	55
N/A	TW1/0/11:3	22	56
H1/0/12	TW1/0/11:4	23	57
H1/0/13	TW1/0/13:1	24	58
N/A	TW1/0/13:2	25	59
N/A	TW1/0/13:3	26	60
H1/0/14	TW1/0/13:4	27	61
H1/0/15	TW1/0/15:1	28	62
N/A	TW1/0/15:2	29	63
N/A	TW1/0/15:3	30	64
H1/0/16	TW1/0/15:4	31	65
H1/0/17	TW1/0/17:1	32	68
N/A	TW1/0/17:2	33	69
N/A	TW1/0/17:3	34	70
H1/0/18	TW1/0/17:4	35	71
H1/0/19	TW1/0/19:1	36	72
N/A	TW1/0/19:2	37	73
N/A	TW1/0/19:3	38	74
H1/0/20	TW1/0/19:4	39	75
H1/0/21	TW1/0/21:1	40	76
N/A	TW1/0/21:2	41	77
N/A	TW1/0/21:3	42	78
H1/0/22	TW1/0/21:4	43	79
H1/0/23	TW1/0/23:1	44	80
N/A	TW1/0/23:2	45	81
N/A	TW1/0/23:3	46	82
H1/0/24	TW1/0/23:4	47	83
H1/0/25	TW1/0/25:1	48	84
N/A	TW1/0/25:2	49	85
N/A	TW1/0/25:3	50	86
H1/0/26	TW1/0/25:4	51	87
H1/0/27	TW1/0/27:1	52	88
N/A	TW1/0/27:2	53	89
N/A	TW1/0/27:3	54	90
H1/0/28	TW1/0/27:4	55	91
H1/0/29	TW1/0/29:1	56	92
N/A	TW1/0/29:2	57	93
N/A	TW1/0/29:3	58	94
H1/0/30	TW1/0/29:4	59	95
H1/0/31	TW1/0/31:1	60	96
N/A	TW1/0/31:2	61	97
N/A	TW1/0/31:3	62	98
H1/0/32	TW1/0/31:4	63	99
H1/0/33	TW1/0/33:1	64	1
N/A	TW1/0/33:2	65	2
N/A	TW1/0/33:3	66	3
H1/0/34	TW1/0/33:4	67	4
H1/0/35	TW1/0/35:1	68	5
N/A	TW1/0/35:2	69	6
N/A	TW1/0/35:3	70	7
H1/0/36	TW1/0/35:4	71	8
H1/0/37	TW1/0/37:1	72	9
N/A	TW1/0/37:2	73	10
N/A	TW1/0/37:3	74	11
H1/0/38	TW1/0/37:4	75	12
H1/0/39	TW1/0/39:1	76	13
N/A	TW1/0/39:2	77	14
N/A	TW1/0/39:3	78	15
H1/0/40	TW1/0/39:4	79	16
H1/0/41	TW1/0/41:1	80	17
N/A	TW1/0/41:2	81	18
N/A	TW1/0/41:3	82	19
H1/0/42	TW1/0/41:4	83	20
H1/0/43	TW1/0/43:1	84	21
N/A	TW1/0/43:2	85	22
N/A	TW1/0/43:3	86	23
H1/0/44	TW1/0/43:4	87	24
H1/0/45	TW1/0/45:1	88	25
N/A	TW1/0/45:2	89	26
N/A	TW1/0/45:3	90	27
H1/0/46	TW1/0/45:4	91	28
H1/0/47	TW1/0/47:1	92	29
N/A	TW1/0/47:2	93	30
N/A	TW1/0/47:3	94	31
H1/0/48	TW1/0/47:4	95	32
H1/0/49	TW1/0/49:1	96	102
N/A	TW1/0/49:2	97	103
N/A	TW1/0/49:3	98	104
H1/0/50	TW1/0/49:4	99	105
H1/0/51	TW1/0/51:1	100	106
N/A	TW1/0/51:2	101	107
N/A	TW1/0/51:3	102	108
H1/0/52	TW1/0/51:4	103	109
H1/0/53	TW1/0/53:1	104	110
N/A	TW1/0/53:2	105	111
N/A	TW1/0/53:3	106	112
H1/0/54	TW1/0/53:4	107	113
H1/0/55	TW1/0/55:1	108	114
N/A	TW1/0/55:2	109	115
N/A	TW1/0/55:3	110	116
H1/0/56	TW1/0/55:4	111	117
H1/0/57	TW1/0/57:1	112	118
N/A	TW1/0/57:2	113	119
N/A	TW1/0/57:3	114	120
H1/0/58	TW1/0/57:4	115	121
H1/0/59	TW1/0/59:1	116	122
N/A	TW1/0/59:2	117	123
N/A	TW1/0/59:3	118	124
H1/0/60	TW1/0/59:4	119	125
H1/0/61	TW1/0/61:1	120	126
N/A	TW1/0/61:2	121	127
N/A	TW1/0/61:3	122	128
H1/0/62	TW1/0/61:4	123	129
H1/0/63	TW1/0/63:1	124	130
N/A	TW1/0/63:2	125	131
N/A	TW1/0/63:3	126	132
H1/0/64	TW1/0/63:4	127	133

You can configure only one sampling feature on the device. Sampling-based mirroring, NetStream (including NetStream and IPv6 NetStream), sFlow, INT, telemetry stream, and MOD features are mutually exclusive. For more information about mirroring, NetStream, IPv6 NetStream, and sFlow, see the network management and monitoring configuration guide of the device. For more information about telemetry stream and MOD, see the telemetry configuration guide of the device.

INT and telemetry stream are mutually exclusive regardless of whether sampling is configured.

When an S9820-8C switch enabled with INT communicates with an S6805, S6825, S6850, or S9850 switch, you must specify the timestamp format inserted into INT packets by executing the telemetry ifa timestamp-format ts command in system view.

Telemetry stream usage guidelines

Telemetry stream can only be deployed on underlay networks.

When you use telemetry stream to collect timestamps reported by multiple nodes to calculate the forwarding path delay, you must configure NTP or PTP for time synchronization on each device.

Configuring telemetry stream requires service loopback interfaces.

You can configure only one sampling feature on the device. Sampling-based mirroring, NetStream (including NetStream and IPv6 NetStream), sFlow, INT, telemetry stream, and MOD features are mutually exclusive. For more information about mirroring, NetStream, IPv6 NetStream, and sFlow, see the network management and monitoring configuration guide of the device. For more information about INT and MOD, see the telemetry configuration guide of the device.

INT and telemetry stream are mutually exclusive regardless of whether sampling is configured.

MOD usage guidelines

MOD monitors only packets dropped for a reason on the configured packet drop reason list. MOD does not monitor packets dropped due to other reasons.

To have the flow group feature take effect correctly when you configure a flow group in MOD mode, you must first change the hardware resource mode to EM mode for the MAC table, ARP table, ND table, and routing table of the device. For more information about the hardware-resource switch-mode command, see device management in the fundamentals command reference of the device. You do not need to change the hardware resource mode to EM mode when you configure a flow group in simple MOD mode.

A sampling feature is required for the MOD mode and is not required for the simple MOD mode.

TCB usage guidelines

The TCB configuration in system view takes effect on all TCB-capable interfaces. The TCB configuration in interface view takes effect only on the specified interface. You can configure TCB in system view or interface view, but not in both views at the same time. For example, if you have configured TCB in system view, to configure TCB in interface view, you must first delete the TCB configuration in system view.

When you configure TCB in interface view, you can configure TCB for only one interface on the device.

The gRPC sampling paths for TCB are tcb/tcbpacketinfoevent and tcb/tcbrawpacketinfoevent.

When you enable TCB on a VTEP in the VXLAN network to monitor dropped packets sent from the VTEP to the tenant network, the characteristics of a dropped packet obtained by TCB is the outer header information of the VXLAN packet instead of the inner header information.

Visualization configuration examples

RDMA network visualization configuration example

Network configuration

As shown in Figure 11, the RDMA network uses the two-tier spine-leaf architecture. The spine devices are S9850-32H switches, and the leaf devices are S6850-56HF devices. The leaf devices act as the gateways of the servers. A Layer 3 ECMP network is built between spine devices and leaf devices. The Layer 3 ECMP network forwards data at a high speed and also supports redundancy of forwarding paths.

To better monitor the cache usage and the forwarding path delay, make sure the following requirements are met:

· Implement lossless transmission of RDMA application packets (with DSCP value 46).

· Configure gRPC to report queue congestion statistics, egress packet drop information, PFC flow control frame statistics, and ECN marking statistics to the gRPC server.

· To monitor packet forwarding paths and link delay, configure INT to add INT marks to packets sent from Host A to Host C. Configure Device E to encapsulate INT packets and then send them to the collector.

NOTE:

This example uses two spine devices and three leaf devices. In actual applications, you can deploy six S9850-32H switches on the spine tier and 32 S6850-56HF switches on the leaf tier to build a low-cost and small-scale RDMA network with the convergence ratio of 2:1.

Figure 11 Network diagram

Analysis

RDMA network deployment

To losslessly transmit RDMA application packets, configure PFC and ECN:

· PFC performs flow control for packets based on their priority queues. RDMA packets carry DSCP value 46. Map DSCP value 46 to 802.1P priority 5, and enable PFC for packets with 802.1P priority 5.

PFC must be enabled on all interfaces along the RDMA packet forwarding paths. Therefore, enable PFC on all interfaces connecting spine devices to leaf devices and interfaces connecting leaf devices to hosts.

· ECN provides end-to-end congestion control. After congestion is detected on the device, the ECN field is marked for related packets. When the receiver receives packets with the ECN field marked, the receiver sends the congestion notification packets to the sender to notify the sender to decrease the packet sending rate. In this example, when the traffic received by a leaf device is greater than the uplink bandwidth, congestion will occur on the port that interconnects the leaf device and spine device. Enable ECN on such ports.

To have ECN take effect first, the upper limit of the average queue length for ECN (configured by using the queue queue-id [ drop-level drop-level ] low-limit low-limit high-limit high-limit [ discard-probability discard-prob ] command) must be less than the PFC back pressure frame triggering threshold. In this example, the back pressure frame triggering threshold is the default static back pressure frame triggering threshold, which is 512. The example sets the upper limit of the average queue length for ECN to 480.

Configuring visualization

To obtain the buffer usage of each device and avoid packet dropping, configure gRPC to subscribe to the following information for analysis and processing of the NMS.

· Periodically reported buffer usage, headroom usage, ECN marking statistics, and PFC flow control frame sending and receiving statistics.

· Reported real-time queue buffer usage violation events and packet dropping events occurring in queues.

To obtain the packet forwarding paths and delay, configure INT to monitor the packets sent from Host A to Host C. Two forwarding paths exist between Host A and Host C: Host A → Device C → Device A → Device E → Host C and Host A → Device C → Device B → Device E → Host C. Configure Device C as the entry node, Device A and Device B as transit nodes, and Device E as the exit node.

Restrictions and guidelines

For PFC to operate correctly on an interface, follow these guidelines when configuring PFC:

· Make sure the interface is configured to trust the 802.1p priority or DSCP value carried in packets by using the qos trust { dot1p | dscp } command.

· Make sure the 802.1p-local priority mapping and DSCP-802.1p priority mapping are the same on all interfaces along the forwarding path.

When you configure gRPC to subscribe to data, follow these restrictions and guidelines:

· Specify the data sampling interval (sample-interval) only for periodic sensor paths.

· If you specify the option for event-triggered sensor paths, the sensor paths do not take effect.

· If you do not specify the option for periodic sensor paths, the device does not sample or push data.

As a best practice to configure INT, first configure the transit nodes and exit node, and then configure the entry node.

Procedure

Assign IP addresses to interfaces

· Configure Device A:

# Set the link mode of HundredGigE 1/0/1 through HundredGigE 1/0/3 to Layer 3, and assign an IP address to each interface.

[DeviceA] interface hundredgige 1/0/1

[DeviceA-HundredGigE1/0/1] port link-mode route

[DeviceA-HundredGigE1/0/1] ip address 10.1.2.2 24

[DeviceA-HundredGigE1/0/1] quit

[DeviceA] interface hundredgige 1/0/2

[DeviceA-HundredGigE1/0/2] port link-mode route

[DeviceA-HundredGigE1/0/2] ip address 20.1.2.2 24

[DeviceA-HundredGigE1/0/2] quit

[DeviceA] interface hundredgige 1/0/3

[DeviceA-HundredGigE1/0/3] port link-mode route

[DeviceA-HundredGigE1/0/3] ip address 30.1.2.2 24

[DeviceA-HundredGigE1/0/3] quit

· Configure Device B:

Set the link mode of each device interface to Layer 3, and assign an IP address to each interface as shown in Figure 11. (Details not shown.)

· Configure Device C:

# Create VLAN 10, assign Twenty-FiveGigE 1/0/1 to VLAN 10, and then assign an IP address to VLAN-interface 10.

<DeviceC> system-view

[DeviceC] vlan 10

[DeviceC-vlan10] port twenty-fivegige 1/0/1

[DeviceC-vlan10] quit

[DeviceC] interface vlan-interface 10

[DeviceC-Vlan-interface10] ip address 10.1.1.1 24

[DeviceC-Vlan-interface10] quit

# Set the link mode of HundredGigE 1/0/25 and HundredGigE 1/0/26 to Layer 3, and assign an IP address to each interface.

[DeviceC] interface hundredgige 1/0/25

[DeviceC-HundredGigE1/0/25] port link-mode route

[DeviceC-HundredGigE1/0/25] ip address 10.1.2.1 24

[DeviceC-HundredGigE1/0/25] quit

[DeviceC] interface hundredgige 1/0/26

[DeviceC-HundredGigE1/0/26] port link-mode route

[DeviceC-HundredGigE1/0/26] ip address 10.1.3.1 24

[DeviceC-HundredGigE1/0/26] quit

· Configure Device D and Device E:

Configure these devices in the same way Device C is configured. Configure the devices based on the VLANs and IP addresses as shown in Figure 11. (Details not shown.)

Enable local proxy ARP and broadcast, multicast, and unknown unicast storm suppression for user-facing interfaces on leaf devices

· Configure Device C:

# Configure broadcast, multicast, and unknown unicast storm suppression on Twenty-FiveGigE 1/0/1 to prevent the interface from forwarding too much flooding traffic and avoid impact on traffic forwarding.

[DeviceC] interface twenty-fivegige 1/0/1

[DeviceC-Twenty-FiveGigE1/0/1] broadcast-suppression pps 0

[DeviceC-Twenty-FiveGigE1/0/1] multicast-suppression pps 0

[DeviceC-Twenty-FiveGigE1/0/1] unicast-suppression pps 0

[DeviceC-Twenty-FiveGigE1/0/1] quit

# Enable local proxy ARP on VLAN-interface 10. Then, the hosts on the same subnet can communicate with each other.

[DeviceC] interface vlan-interface 10

[DeviceC-Vlan-interface10] local-proxy-arp enable

· Configure Device D and Device E:

Configure these devices in the same way Device C is configured. (Details not shown.)

Configure the routing protocol

· Configure Device A:

# Configure OSPF to advertise subnet routes for the interfaces.

[DeviceA] ospf 1 router-id 1.1.1.1

[DeviceA-ospf-1] area 0

[DeviceA-ospf-1-area-0.0.0.0] network 10.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] network 20.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] network 30.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] quit

[DeviceA-ospf-1] quit

· Configure Device B:

# Configure OSPF to advertise subnet routes for the interfaces.

<DeviceB> system-view

[DeviceB] ospf 1 router-id 2.2.2.2

[DeviceB-ospf-1] area 0

[DeviceB-ospf-1-area-0.0.0.0] network 10.1.3.0 0.0.0.255

[DeviceB-ospf-1-area-0.0.0.0] network 20.1.3.0 0.0.0.255

[DeviceB-ospf-1-area-0.0.0.0] network 30.1.3.0 0.0.0.255

[DeviceB-ospf-1-area-0.0.0.0] quit

[DeviceB-ospf-1] quit

· Configure Device C:

# Configure OSPF to advertise subnet routes for the interfaces.

<DeviceC> system-view

[DeviceC] ospf 1 router-id 3.3.3.3

[DeviceC-ospf-1] area 0

[DeviceC-ospf-1-area-0.0.0.0] network 10.1.1.0 0.0.0.255

[DeviceC-ospf-1-area-0.0.0.0] network 10.1.2.0 0.0.0.255

[DeviceC-ospf-1-area-0.0.0.0] network 10.1.3.0 0.0.0.255

[DeviceC-ospf-1-area-0.0.0.0] quit

[DeviceC-ospf-1] quit

· Configure Device D:

# Configure OSPF to advertise subnet routes for the interfaces.

<DeviceD> system-view

[DeviceD] ospf 1 router-id 4.4.4.4

[DeviceD-ospf-1] area 0

[DeviceD-ospf-1-area-0.0.0.0] network 20.1.1.0 0.0.0.255

[DeviceD-ospf-1-area-0.0.0.0] network 20.1.2.0 0.0.0.255

[DeviceD-ospf-1-area-0.0.0.0] network 20.1.3.0 0.0.0.255

[DeviceD-ospf-1-area-0.0.0.0] quit

[DeviceD-ospf-1] quit

· Configure Device E:

# Configure OSPF to advertise subnet routes for the interfaces.

<DeviceE> system-view

[DeviceE] ospf 1 router-id 5.5.5.5

[DeviceE-ospf-1] area 0

[DeviceE-ospf-1-area-0.0.0.0] network 30.1.1.0 0.0.0.255

[DeviceE-ospf-1-area-0.0.0.0] network 30.1.2.0 0.0.0.255

[DeviceE-ospf-1-area-0.0.0.0] network 30.1.3.0 0.0.0.255

[DeviceE-ospf-1-area-0.0.0.0] quit

[DeviceE-ospf-1] quit

Configure PFC and PFC deadlock detection

· Configure Device A:

# Configure the DSCP-to-802.1p priority mapping table to map DSCP value 46 to 802.1p priority 5.

[DeviceA] qos map-table dscp-dot1p

[DeviceA-maptbl-dot1p-dot1p] import 46 export 5

[DeviceA-maptbl-dot1p-dot1p] quit

# On interfaces HundredGigE 1/0/1, HundredGigE 1/0/2, and HundredGigE 1/0/3, configure trusting DSCP values in packets, enable PFC, and enable PFC for 802.1p priority 5.

[DeviceA] interface range hundredgige 1/0/1 to hundredgige 1/0/3

[DeviceA-if-range] qos trust dscp

[DeviceA-if-range] priority-flow-control enable

[DeviceA-if-range] priority-flow-control no-drop dot1p 5

# On interfaces HundredGigE 1/0/1, HundredGigE 1/0/2, and HundredGigE 1/0/3, set the PFC deadlock detection interval to 10, set the recovery mode for PFC deadlock detection to auto (default setting), and enable PFC deadlock detection.

[DeviceA-if-range] priority-flow-control deadlock cos 5 interval 10

[DeviceA-if-range] priority-flow-control deadlock auto-recover cos 5 delay 10

[DeviceA-if-range] priority-flow-control deadlock recover-mode auto

[DeviceA-if-range] priority-flow-control deadlock enable

[DeviceA-if-range] quit

· Configure Device B:

Configure Device B in the same way Device A is configured. (Details not shown.)

· Configure Device C:

# Configure the DSCP-to-802.1p priority mapping table to map DSCP value 46 to 802.1p priority 5.

[DeviceC] qos map-table dscp-dot1p

[DeviceC-maptbl-dot1p-dot1p] import 46 export 5

[DeviceC-maptbl-dot1p-dot1p] quit

# On interfaces Twenty-FiveGigE 1/0/1, HundredGigE 1/0/25, and HundredGigE 1/0/26, configure trusting DSCP values in packets, enable PFC, and enable PFC for 802.1p priority 5.

[DeviceC] interface range twenty-fivegige 1/0/1 hundredgige 1/0/25 to hundredgige 1/0/26

[DeviceC-if-range] qos trust dscp

[DeviceC-if-range] priority-flow-control enable

[DeviceC-if-range] priority-flow-control no-drop dot1p 5

# On interfaces Twenty-FiveGigE 1/0/1, HundredGigE 1/0/25, and HundredGigE 1/0/26, set the PFC deadlock detection interval to 10, set the recovery mode for PFC deadlock detection to auto (default setting), and enable PFC deadlock detection.

[DeviceC-if-range] priority-flow-control deadlock cos 5 interval 10

[DeviceC-if-range] priority-flow-control deadlock auto-recover cos 5 delay 10

[DeviceC-if-range] priority-flow-control deadlock recover-mode auto

[DeviceC-if-range] priority-flow-control deadlock enable

[DeviceC-if-range] quit

· Configure Device D and Device E:

Configure these devices in the same way Device C is configured. (Details not shown.)

Configure ECN

· Configure Device A:

# Create WRED table queue-table5. In the WRED table, set the lower limit for the average queue length to 100, the upper limit for the average queue length to 480, and the drop probability to 20%. Enable ECN for queue 5. Apply WRED table queue-table5 to interfaces HundredGigE 1/0/1 through HundredGigE 1/0/3.

[DeviceA] qos wred queue table queue-table5

[DeviceA-wred-table-queue-table5] queue 5 low-limit 100 high-limit 480 discard-probability 20

[DeviceA-wred-table-queue-table5] queue 5 ecn

[DeviceA-wred-table-queue-table5] quit

[DeviceA] interface range hundredgige 1/0/1 to hundredgige 1/0/3

[DeviceA-if-range] qos wred apply queue-table5

[DeviceA-if-range] quit

· Configure Device B:

Configure Device B in the same way Device A is configured. (Details not shown.)

· Configure Device C:

[DeviceC] qos wred queue table queue-table5

[DeviceC-wred-table-queue-table5] queue 5 low-limit 100 high-limit 480 discard-probability 20

[DeviceC-wred-table-queue-table5] queue 5 ecn

[DeviceC-wred-table-queue-table5] quit

[DeviceC] interface range hundredgige 1/0/25 to hundredgige 1/0/26

[DeviceC-if-range] qos wred apply queue-table5

[DeviceC-if-range] quit

· Configure Device D and Device E:

Configure these devices in the same way Device C is configured. (Details not shown.)

Configure gRPC

· Configure Device A:

# Enable threshold-crossing alarms for the egress buffer.

[DeviceA] buffer threshold alarm egress enable

# Enable threshold-crossing alarms for the headroom buffer.

[DeviceA] buffer threshold alarm headroom enable

# Enable packet-drop alarms.

[DeviceA] buffer packet-drop alarm enable

# Enable gRPC.

[DeviceA] grpc enable

# Create sensor group test1, and add sensor paths bufferusages, headroomusages, egressdrops, pfcstatistics, and ecnandwredstatistics.

[DeviceA] telemetry

[DeviceA-telemetry] sensor-group test1

[DeviceA-telemetry-sensor-group-test1] sensor path buffermonitor/bufferusages

[DeviceA-telemetry-sensor-group-test1] sensor path buffermonitor/headroomusages

[DeviceA-telemetry-sensor-group-test1] sensor path buffermonitor/egressdrops

[DeviceA-telemetry-sensor-group-test1] sensor path buffermonitor/pfcstatistics

[DeviceA-telemetry-sensor-group-test1] sensor path buffermonitor/ecnandwredstatistics

[DeviceA-telemetry-sensor-group-test1] quit

# Create sensor group test2, and add sensor paths portqueoverrunevent and portquedropevent.

[DeviceA-telemetry] sensor-group test2

[DeviceA-telemetry-sensor-group-test2] sensor path buffermonitor/portqueoverrunevent

[DeviceA-telemetry-sensor-group-test2] sensor path buffermonitor/portquedropevent

[DeviceA-telemetry-sensor-group-test2] quit

# Create destination group collector1, and add a collector that uses IPv4 address 40.1.1.1 and port number 50050 to the destination group. (Make sure Device A and the collector can reach each other. Details are not shown.)

[DeviceA-telemetry] destination-group collector1

[DeviceA-telemetry-destination-group-collector1] ipv4-address 40.1.1.1 port 50050

[DeviceA-telemetry-destination-group-collector1] quit

# Create subscription A, specify sensor group test1 for the subscription with the data sampling interval 10 seconds, and specify sensor group test2 and destination group collector1 for the subscription.

[DeviceA-telemetry] subscription A

[DeviceA-telemetry-subscription-A] sensor-group test1 sample-interval 10

[DeviceA-telemetry-subscription-A] sensor-group test2

[DeviceA-telemetry-subscription-A] destination-group collector1

[DeviceA-telemetry-subscription-A] quit

· Configure Device B, Device C, Device D, and Device E:

Configure these devices in the same way Device A is configured. (Details not shown.)

Configure INT

· Configure Device E (exit node):

# Specify IPv4 address 30.1.1.1 as the device ID.

[DeviceE] telemetry ifa device-id 30.1.1.1

# Specify input interfaces HundredGigE 1/0/25 and HundredGigE 1/0/26 as egress interfaces.

[DeviceE] interface range hundredgige 1/0/25 to hundredgige 1/0/26

[DeviceE-if-range] telemetry ifa role egress

[DeviceE-if-range] quit

# Configure addressing parameters to encapsulate in INT packets sent to the collector.

[DeviceE] telemetry ifa collector source 30.1.1.1 destination 40.1.1.1 source-port 12 destination-port 14

# Enable INT.

[DeviceE] telemetry ifa global enable

· Configure Device A (transit node):

# Specify the device ID as IPv4 address 10.1.2.2.

[DeviceA] telemetry ifa device-id 10.1.2.2

# Specify input interface HundredGigE1/0/1 as the transit interface.

[DeviceA] interface hundredgige 1/0/1

[DeviceA-HundredGigE1/0/1] telemetry ifa role transit

[DeviceA-HundredGigE1/0/1] quit

# Enable INT.

[DeviceA] telemetry ifa global enable

· Configure Device B (transit node):

Configure Device B in the same way Device A is configured except the device ID. Specify IPv4 address 10.1.3.2 as the device ID. (Details not shown.)

· Configure Device C (entry node):

# Specify the device ID as IPv4 address 10.1.1.1.

[DeviceC] telemetry ifa device-id 10.1.1.1

# Create sampler samp, set the sampling mode to random, and set the sampling rate to 2 to the 8th power.

[DeviceC] sampler samp mode random packet-interval n-power 8

# Create IPv4 basic ACL 2000, and configure a rule to match packets with source IP address 10.1.1.2 in the ACL.

[DeviceC] acl basic 2000

[DeviceC-acl-ipv4-basic-2000] rule permit source 10.1.1.2 0

[DeviceC-acl-ipv4-basic-2000] quit

# Specify input interface Twenty-FiveGigE 1/0/1 as the ingress interface.

[DeviceC] interface twenty-fivegige 1/0/1

[DeviceC-Twenty-FiveGigE1/0/1] telemetry ifa role ingress

[DeviceC-Twenty-FiveGigE1/0/1] quit

# Create traffic class classifier1 to match basic ACL 2000.

[DeviceC] traffic classifier classifier1

[DeviceC-classifier-classifier1] if-match acl 2000

[DeviceC-classifier-classifier1] quit

# Create traffic behavior behavior1, and configure the action of mirroring traffic to the INT processor.

[DeviceC] traffic behavior behavior1

[DeviceC-behavior-behavior1] mirror-to ifa-processor sampler samp

[DeviceC-behavior-behavior1] quit

# Create a QoS policy named ifa1, and associate traffic class classifier1 with traffic behavior behavior1 in the QoS policy.

[DeviceC] qos policy ifa1

[DeviceC-qospolicy-ifa1] classifier classifier1 behavior behavior1

[DeviceC-qospolicy-ifa1] quit

# Apply QoS policy ifa1 to the incoming traffic of Twenty-FiveGigE 1/0/1.

[DeviceC] interface twenty-fivegige 1/0/1

[DeviceC-Twenty-FiveGigE1/0/1] qos apply policy ifa1 inbound

[DeviceC-Twenty-FiveGigE1/0/1] quit

# Enable INT.

[DeviceC] telemetry ifa global enable

Verifying the configuration

When the 10 servers connected to Device C (including Host A) simultaneously forward traffic to Host C at line speed, the following results occur:

· The gRPC collector can receive ECN marking statistics, egress buffer usage violation events, and PFC flow control frame sending and receiving statistics of HundredGigE 1/0/25 and HundredGigE 1/0/26 reported by Device C.

· The server reduces the packet sending rate, stops packet sending for a short period of time, and then resumes packet sending.

· The INT collector can receive INT packets sent by Device E to analyze the forwarding paths and delay.

MOD configuration example

Network configuration

Figure 12 Network diagram

Analysis

· To generate flow entries based on the flow group, configure the flow group mode to MOD.

· Based on the packet drop reason list of MOD, the device monitors packet drops for packets matching the flow entries.

· Configure MOD sampling to send sampled packets to the collector through UDP.

Restrictions and guidelines

· You can specify only one IPv4 ACL for a flow group. The ACL only supports matching the 5-tuple (the source IP address, destination IP address, source port number, destination port number, and protocol type) and DSCP value of packets.

· Because a flow can belong to only one flow group, make sure the same flow is not assigned to more than one flow group when specifying ACLs.

· MOD takes effect only on packets matching ACLs specified for flow groups. MOD monitors only packets dropped for a reason on the configured packet drop reason list. MOD does not monitor packets dropped due to other reasons.

· To delete an applied flow group, first remove the application and then delete the flow group.

· Only flow groups in the same mode can be enabled simultaneously on the device.

· You cannot edit the name or mode of an existing flow group.

Procedure

Prerequisites

Before you configure MOD, make sure the hardware resource mode of the device is the EM mode. To view the current hardware resource mode of the device, execute the display hardware-resource switch-mode command. Perform the following procedure to change the hardware resource mode to EM mode.

1. Configure the hardware resource mode as EM mode. Save the configuration and restart the system to have the EM mode take effect.

[DeviceA] hardware-resource switch-mode EM

Do you want to change the specified hardware resource working mode? [Y/N]:y

The hardware resource working mode is changed, please save the configuration and reboot the system to make it effective.

2. Repeat the operation on Device B, Device C, Device D, and Device E. (Details not shown.)

Configure Device A

· Configure IP addresses for interfaces:

# Set the link mode of HundredGigE 1/0/1 through HundredGigE 1/0/3 to Layer 3, and assign an IP address to each interface.

<DeviceA> system-view

[DeviceA] interface hundredgige 1/0/1

[DeviceA-HundredGigE1/0/1] port link-mode route

[DeviceA-HundredGigE1/0/1] ip address 10.1.2.2 24

[DeviceA-HundredGigE1/0/1] quit

[DeviceA] interface hundredgige 1/0/2

[DeviceA-HundredGigE1/0/2] port link-mode route

[DeviceA-HundredGigE1/0/2] ip address 20.1.2.2 24

[DeviceA-HundredGigE1/0/2] quit

[DeviceA] interface hundredgige 1/0/3

[DeviceA-HundredGigE1/0/3] port link-mode route

[DeviceA-HundredGigE1/0/3] ip address 30.1.2.2 24

[DeviceA-HundredGigE1/0/3] quit

· Configure the routing protocol:

# Configure OSPF to advertise subnet routes for the interfaces.

[DeviceA] ospf 1 router-id 1.1.1.1

[DeviceA-ospf-1] area 0

[DeviceA-ospf-1-area-0.0.0.0] network 10.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] network 20.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] network 30.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] quit

[DeviceA-ospf-1] quit

· Configure a flow group:

# Create IPv4 advanced ACL 3000, and configure a rule to match packets with destination IP address 30.1.1.2 in the ACL.

<DeviceA> system-view

[DeviceA] acl advanced 3000

[DeviceA-acl-ipv4-adv-3000] rule permit ip destination 30.1.1.2 0

[DeviceA-acl-ipv4-adv-3000] quit

# Create flow group 1 in MOD mode and specify ACL 3000 for it.

[DeviceA] telemetry flow-group 1

[DeviceA-telemetry-flow-group-1] if-match acl 3000

# Configure the flow group to generate a flow entry based on the destination IP address.

[DeviceA-telemetry-flow-group-1] template destination-ip

[DeviceA-telemetry-flow-group-1] quit

# Set the flow entry aging timer to 10 minutes.

[DeviceA] telemetry flow-group aging-time 10

# Apply flow group 1.

[DeviceA] telemetry apply flow-group 1

· Configure MOD:

# Configure the device ID for MOD as 10.1.2.2.

[DeviceA] telemetry mod

[DeviceA-telemetry-mod] device-id 10.1.2.2

[DeviceA-telemetry-mod] quit

# Create sampler samp, set the sampling mode to random, and set the sampling rate to 2 to the 8th power.

[DeviceA] sampler samp mode random packet-interval n-power 8

# Enable sampling for MOD, and reference sampler samp.

[DeviceA] telemetry mod

[DeviceA-telemetry-mod] sampler samp

# Configure MOD to use UDP to send packets to the collector.

[DeviceA-telemetry-mod] transport-protocol udp

# Encapsulate the packets sent to the collector by MOD with the following information: source IP address 10.1.2.2, destination IP address 40.1.1.1, source port number 1000, and destination port number 2333.

[DeviceA-telemetry-mod] collector source-ip 10.1.2.2 destination-ip 40.1.1.1 source-port 1000 destination-port 2333

# Configure MOD to monitor all reasons for dropping packets.

[DeviceA-telemetry-mod] reason-list all

[DeviceA-telemetry-mod] quit

Configure Device B, Device C, Device D, and Device E

Configure Device B, Device C, Device D, and Device E in the same way Device A is configured except the interface IP addresses, OSPF routes, device ID for MOD, and the source IP address encapsulated in the packets sent to the collector. (Details not shown.)

Verifying the configuration

Displaying configuration on Device A

# Display the ACL configuration.

<DeviceA> display acl 3000

Advanced IPv4 ACL 3000, 1 rule,

ACL's step is 5, start ID is 0

rule 0 permit ip destination 30.1.1.2 0

# Display the flow group configuration.

<DeviceA> display telemetry flow-group 1

Flow group 1 (Successful)

ACL : 3000

Template : destination-ip

Mode : MOD

Aging time: 10 minutes

Rate limit： -

Max-entry ： -

# Display the MOD configuration.

<DeviceA> display telemetry mod

Status :Successful

Drop reason list:

ipv4-dip-miss

ip-multicase-error

unknown-vlan

ipv4-l3-header-error

tunnel-header-error

parity-error

higig-header-error

invalid-tpid

Sampler : samp

Device ID : 10.1.2.2

Transport protocol : UDP

Collector

Source IP : 10.1.2.2

Destination IP : 40.1.1.1

Source port : 1000

Destination port : 2333

Displaying configuration on Device B, Device C, Device D, and Device E

Analyzing MOD packets received by the collector

# Ping Host C from Device A.

<DeviceA> ping 30.1.1.2

Ping 30.1.1.2 (30.1.1.2): 56 data bytes, press CTRL+C to break

56 bytes from 30.1.1.2: icmp_seq=0 ttl=255 time=4.703 ms

56 bytes from 30.1.1.2: icmp_seq=1 ttl=255 time=1.636 ms

56 bytes from 30.1.1.2: icmp_seq=2 ttl=255 time=1.733 ms

56 bytes from 30.1.1.2: icmp_seq=3 ttl=255 time=1.662 ms

56 bytes from 30.1.1.2: icmp_seq=4 ttl=255 time=1.606 ms

--- Ping statistics for 30.1.1.2 ---

5 packet(s) transmitted, 5 packet(s) received, 0.0% packet loss

round-trip min/avg/max/std-dev = 1.606/2.268/4.703/1.218 ms

# Edit configuration of Device E.

<DeviceE> system-view

[DeviceE] ospf 1

[DeviceE-ospf-1] area 0

[DeviceE-ospf-1-area-0.0.0.0] undo network 30.1.2.0 0.0.0.255

[DeviceE-ospf-1-area-0.0.0.0] quit

[DeviceE-ospf-1] quit

# Ping Host C from Device A.

<DeviceA> ping 30.1.1.2

Ping 30.1.1.2 (30.1.1.2): 56 data bytes, press CTRL+C to break

Request time out

--- Ping statistics for 30.1.1.2 ---

5 packet(s) transmitted, 0 packet(s) received, 100.0% packet loss

When a packet destined for Host C is dropped during the forwarding process of a spine device or leaf device, the collector can receive the packet drop reason and the characteristics of the dropped packet from the device.

In this example, the packet drop information received by the collector shows that Device A has dropped packets and the packet drop reason is ipv4-dip-miss. This indicates that Device A does not have any route to Host C. After the OSPF configuration on Device E is restored, Host C can receive packets from other devices.

Simple MOD configuration example

Network configuration

As shown in Figure 12, the network uses a two-tier spine-leaf architecture. Configure MOD on the spine and leaf devices to identify whether packets are dropped during the forwarding process on the devices. MOD sends the collected packet drop information to the collector for analysis and processing. Configure simple MOD to save hardware entry resource.

Figure 13 Network diagram

Analysis

· To generate flow entries based on the flow group, configure the flow group mode to simple MOD.

· Based on the packet drop reason list of simple MOD, the device monitors packet drops for packets matching the flow entries.

Restrictions and guidelines

· Because a flow can belong to only one flow group, make sure the same flow is not assigned to more than one flow group when specifying ACLs.

· To delete an applied flow group, first remove the application and then delete the flow group.

· Only flow groups in the same mode can be enabled simultaneously on the device.

· You cannot edit the name or mode of an existing flow group.

Procedure

Configure Device A

· Configure IP addresses for interfaces:

# Set the link mode of HundredGigE 1/0/1 through HundredGigE 1/0/3 to Layer 3, and assign an IP address to each interface.

<DeviceA> system-view

[DeviceA] interface hundredgige 1/0/1

[DeviceA-HundredGigE1/0/1] port link-mode route

[DeviceA-HundredGigE1/0/1] ip address 10.1.2.2 24

[DeviceA-HundredGigE1/0/1] quit

[DeviceA] interface hundredgige 1/0/2

[DeviceA-HundredGigE1/0/2] port link-mode route

[DeviceA-HundredGigE1/0/2] ip address 20.1.2.2 24

[DeviceA-HundredGigE1/0/2] quit

[DeviceA] interface hundredgige 1/0/3

[DeviceA-HundredGigE1/0/3] port link-mode route

[DeviceA-HundredGigE1/0/3] ip address 30.1.2.2 24

[DeviceA-HundredGigE1/0/3] quit

· Configure the routing protocol:

# Configure OSPF to advertise subnet routes for the interfaces.

[DeviceA] ospf 1 router-id 1.1.1.1

[DeviceA-ospf-1] area 0

[DeviceA-ospf-1-area-0.0.0.0] network 10.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] network 20.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] network 30.1.2.0 0.0.0.255

[DeviceA-ospf-1-area-0.0.0.0] quit

[DeviceA-ospf-1] quit

· Configure a flow group:

# Create IPv4 advanced ACL 3000, and configure a rule to match packets with destination IP address 30.1.1.2 in the ACL.

<DeviceA> system-view

[DeviceA] acl advanced 3000

[DeviceA-acl-ipv4-adv-3000] rule permit ip destination 30.1.1.2 0

[DeviceA-acl-ipv4-adv-3000] quit

# Create flow group 2 in simple MOD mode and specify ACL 3000 for it.

[DeviceA] telemetry flow-group 2 mode simple-mod

[DeviceA-telemetry-flow-group-2] if-match acl 3000

# Configure the flow group to generate a flow entry based on the destination IP address.

[DeviceA-telemetry-flow-group-2] template destination-ip

[DeviceA-telemetry-flow-group-2] quit

# Set the flow entry aging timer to 10 minutes.

[DeviceA] telemetry flow-group aging-time 10

# Apply flow group 2.

[DeviceA] telemetry apply flow-group 2

· Configure simple MOD:

# Configure the device ID for simple MOD as 10.1.2.2.

[DeviceA] telemetry mod

[DeviceA-telemetry-mod] device-id 10.1.2.2

# Configure MOD to use UDP to send packets to the collector.

[DeviceA-telemetry-mod] transport-protocol udp

# Encapsulate the packets sent to the collector by simple MOD with the following information: source IP address 10.1.2.2, destination IP address 40.1.1.1, source port number 1000, and destination port number 2333.

[DeviceA-telemetry-mod] collector source-ip 10.1.2.2 destination-ip 40.1.1.1 source-port 1000 destination-port 2333

# Configure MOD to monitor all reasons for dropping packets.

[DeviceA-telemetry-mod] reason-list all

[DeviceA-telemetry-mod] quit

Configure Device B, Device C, Device D, and Device E

Verifying the configuration

Displaying configuration on Device A

# Display the ACL configuration.

<DeviceA> display acl 3000

Advanced IPv4 ACL 3000, 1 rule,

ACL's step is 5, start ID is 0

rule 0 permit ip destination 30.1.1.2 0

# Display the flow group configuration.

<Device> display telemetry flow-group 2

Flow group 2 (Successful)

ACL : 3000

Template : destination-ip

Mode : Simple MOD

Aging time: 10 minutes

Rate limit： -

Max-entry ： -

# Display the MOD configuration.

<DeviceA> display telemetry mod

Status :Successful

Drop reason list:

ipv4-dip-miss

ip-multicase-error

unknown-vlan

tunnel-header-error

parity-error

higig-header-error

Device ID : 10.1.2.2

Transport protocol : UDP

Collector

Source IP : 10.1.2.2

Destination IP : 40.1.1.1

Source port : 1000

Destination port : 2333

Displaying configuration on Device B, Device C, Device D, and Device E

Analyzing MOD packets received by the collector

# Ping Host C from Device A.

<DeviceA> ping 30.1.1.2

Ping 30.1.1.2 (30.1.1.2): 56 data bytes, press CTRL+C to break

56 bytes from 30.1.1.2: icmp_seq=0 ttl=255 time=4.703 ms

56 bytes from 30.1.1.2: icmp_seq=1 ttl=255 time=1.636 ms

56 bytes from 30.1.1.2: icmp_seq=2 ttl=255 time=1.733 ms

56 bytes from 30.1.1.2: icmp_seq=3 ttl=255 time=1.662 ms

56 bytes from 30.1.1.2: icmp_seq=4 ttl=255 time=1.606 ms

--- Ping statistics for 30.1.1.2 ---

5 packet(s) transmitted, 5 packet(s) received, 0.0% packet loss

round-trip min/avg/max/std-dev = 1.606/2.268/4.703/1.218 ms

# Edit configuration of Device E.

<DeviceE> system-view

[DeviceE] ospf 1

[DeviceE-ospf-1] area 0

[DeviceE-ospf-1-area-0.0.0.0] undo network 30.1.2.0 0.0.0.255

[DeviceE-ospf-1-area-0.0.0.0] quit

[DeviceE-ospf-1] quit

# Ping Host C from Device A.

<DeviceA> ping 30.1.1.2

Ping 30.1.1.2 (30.1.1.2): 56 data bytes, press CTRL+C to break

Request time out

--- Ping statistics for 30.1.1.2 ---

5 packet(s) transmitted, 0 packet(s) received, 100.0% packet loss

TCB configuration example

Network configuration

As shown in Figure 14, Device A acts as a gRPC client.

Configure TCB and gRPC in dial-out mode on Device A to monitor packet drops and report packet drop information to the collector through gRPC.

Figure 14 Network diagram

Analysis

· Enable TCB for the egress buffer on Twenty-FiveGigE 1/0/3 of Device A and configure TCB parameters as required to monitor packet drops of the MMU queue.

· Configure gRPC on Device A to send data collected by TCB to the collector. The collector acts as a gRPC server to parse the data.

Restrictions and guidelines

· The TCB configuration in system view takes effect on all TCB-capable interfaces. The TCB configuration in interface view takes effect only on the specified interface. You can configure TCB in system view or interface view, but not in both views at the same time. For example, if you have configured TCB in system view, to configure TCB in interface view, you must first delete the TCB configuration in system view.

· When you configure TCB in interface view, you can configure TCB for only one interface on the device.

· The device sends all packet drop information captured by TCB to the gRPC server, but the gRPC server can only parse packet drop information for IPv4 TCP packets and IPv4 UDP packets.

· The display buffer transient-capture packet-drop command can display a maximum of three packet drop items. This command cannot display the VLAN information of outbound packets that are forwarded between IRF member devices.

Procedure

Prerequisites

Configure IP addresses for devices, and make sure they can reach each other at Layer 3. (Details not shown.)

Configure Device A

· Configure TCB:

# Create IPv4 advanced ACL 3000, and configure a rule to match packets with source IP addresses 192.168.0.1 and 192.168.1.1 in the ACL.

<DeviceA> system-view

[DeviceA] acl advanced 3000

[DeviceA-acl-ipv4-adv-3000] rule permit ip source 192.168.0.1 0

[DeviceA-acl-ipv4-adv-3000] rule permit ip source 192.168.1.1 0

[DeviceA-acl-ipv4-adv-3000] quit

# Enable TCB for packets matching ACL 3000 in the outbound direction of queue 0 on Twenty-FiveGigE 1/0/3 and configure TCB parameters.

[DeviceA] interface twenty-fivegige 1/0/3

[DeviceA-Twenty-FiveGigE1/0/3] buffer transient-capture egress queue 0 enable acl 3000 start-threshold 10000 stop-threshold 5000 frozen-number 1000 frozen-timer 500 pre-sample-rate 10 post-sample-rate 10 poll-frequency 600

[DeviceA-Twenty-FiveGigE1/0/3] quit

· Configure gRPC:

# Enable gRPC.

[DeviceA] grpc enable

# Create a sensor group named test, and add sensor paths tcb/tcbpacketinfoevent and tcb/tcbrawpacketinfoevent to it.

[DeviceA] telemetry

[DeviceA-telemetry] sensor-group test

[DeviceA-telemetry-sensor-group-test] sensor path tcb/tcbpacketinfoevent

[DeviceA-telemetry-sensor-group-test] sensor path tcb/tcbrawpacketinfoevent

[DeviceA-telemetry-sensor-group-test] quit

# Create a destination group named collector1 and add a collector to the group. In this example, the collector receives data on port 10001 at IPv4 address 192.168.2.1.

[DeviceA-telemetry] destination-group collector1

[DeviceA-telemetry-destination-group-collector1] ipv4-address 192.168.2.1 port 10001

[DeviceA-telemetry-destination-group-collector1] quit

# Configure a subscription named A to bind sensor group test with destination group collector1.

[DeviceA-telemetry] subscription A

[DeviceA-telemetry-subscription-A] sensor-group test

[DeviceA-telemetry-subscription-A] destination-group collector1

[DeviceA-telemetry-subscription-A] quit

[DeviceA-telemetry] quit

Verifying the configuration

Send traffic from Server 1 and Server 2 to Server 3 at line speed, and packet drop occurs in the outbound direction of Twenty-FiveGigE 1/0/3 on Device A. The collector can receive packet drop information sent from the device. You can also view the packet drop information captured by TCB on the device.

# Display packet drop information about IPv4 packets captured by TCB for slot 1 on Device A.

<DeviceA> display buffer transient-capture packet-drop slot 1 ipv4

TCB STATUS: EgressPort Twenty-FiveGigE1/0/3

--------------------------------------------------------------------------------

No. Time SMAC DMAC VLAN SIP DIP Type SPort DPort Queue IfName Reason

--------------------------------------------------------------------------------

Total Drop Packets in Ring Buffer of XPE 0:2047

0 178a5cc8eb0 0010-9400-0005 0010-9400-0002 100 192.168.0.1 192.168.3.1

UDP 60 60 2 Twenty-FiveGigE1/0/3 2

Telemetry stream configuration example

Network configuration

As shown in Figure 15, the network uses a two-tier spine-leaf architecture. A delay exists on packets sent from Host A to Host C. Configure telemetry stream on devices along the packet forwarding paths for the collector to analyze the delay.

Figure 15 Network diagram

Analysis

Two forwarding paths exist between Host A and Host C: Host A → Device C → Device A → Device E → Host C and Host A → Device C → Device B → Device E → Host C. Configure telemetry stream on Device C, Device A, Device B, and Device E.

Restrictions and guidelines

An interface assigned to a service loopback group will be exclusively used by the group, and cannot be used for any other purposes. This example uses Twenty-FiveGigE 1/0/5 as an example.

Procedure

Prerequisites

Configure IP addresses for devices, and make sure they can reach each other at Layer 3. (Details not shown.)

Configure Device A

# Specify the ID of Device B as IPv4 address 40.1.2.1.

<DeviceA> system-view

[DeviceA] telemetry stream device-id 40.1.2.1

# Encapsulate the packets sent to the collector with the following information: source IP address 40.1.2.1, destination IP address 40.1.1.1, source port number 12, and destination port number 14.

[DeviceA] telemetry stream collector source 40.1.2.1 destination 40.1.1.1 source-port 12 destination-port 14

# Create service loopback group 1, and specify its service type as telemetry-stream.

[DeviceA] service-loopback group 1 type telemetry-stream

# Assign Twenty-FiveGigE 1/0/5 to service loopback group 1.

[DeviceA] interface twenty-fivegige1/0/5

[DeviceA-Twenty-FiveGigE1/0/5] port service-loopback group 1

All configurations on the interface will be lost. Continue?[Y/N]:y

[DeviceA-Twenty-FiveGigE1/0/5] quit

# Enable telemetry stream timestamping.

[DeviceA] telemetry stream timestamp enable

# Create sampler samp in random sampling mode, and set the sampling rate to 8. One packet from 256 (2 to the 8th power) packets is selected.

[DeviceA] sampler samp mode random packet-interval n-power 8

# Create IPv4 basic ACL 2000, and configure a rule to match packets with source IP address 10.1.1.2 in the ACL.

[DeviceA] acl basic 2000

[DeviceA-acl-ipv4-basic-2000] rule permit source 10.1.1.2

[DeviceA-acl-ipv4-basic-2000] quit

# Configure telemetry stream action 1 to use ACL 2000 and sampler samp on HundredGigE 1/0/1.

[DeviceA] interface hundredgige 1/0/1

[DeviceA-HundredGigE1/0/1] telemetry stream action 1 acl 2000 sampler samp

[DeviceA-HundredGigE1/0/1] quit

# Enable telemetry stream.

[DeviceA] telemetry stream enable

Configure Device B, Device C, Device D, and Device E

Configure Device B, Device C, Device D, and Device E in the same way Device A is configured except the device ID and source IP address encapsulated in the packets sent to the collector. (Details not shown.)

Verifying the configuration

Displaying the telemetry stream configuration on Device A.

[DeviceA] display telemetry stream

Telemetry stream status : Enabled

Telemetry stream timestamp status: Enabled

Telemetry stream device-id : 40.1.2.1

Telemetry stream action:

HundredGigE1/0/1:

Telemetry stream action 1:

ACL : 2000

Sampler: samp

Telemetry stream collector:

Source IP : 40.1.2.1

Destination IP : 40.1.1.1

Source port : 12

Destination port: 14

Displaying configuration on Device B, Device C, Device D, and Device E

Analyzing telemetry stream packets received by the collector

The collector can receive telemetry stream packets that contain the device ID, input interface information, output interface information, input interface timestamp, and output interface timestamp. The information can be used to analyze the delay of packets on each node.

Visualization Technology on H3C DC Switches-6W100

INT packet types

How INT works

Common INT

Flexible INT

Typical network model for visualization

RDMA network deployment

Configuring visualization

Enable local proxy ARP and broadcast, multicast, and unknown unicast storm suppression for user-facing interfaces on leaf devices

Configure the routing protocol

Configure PFC and PFC deadlock detection

Configure ECN

Configure gRPC

Configure INT

Procedure

Verifying the configuration

Simple MOD configuration example

Analysis

Verifying the configuration

TCB configuration example

Restrictions and guidelines

Procedure

Verifying the configuration

Prerequisites

Configure Device A

Configure Device B, Device C, Device D, and Device E

Displaying the telemetry stream configuration on Device A.

Displaying configuration on Device B, Device C, Device D, and Device E

Analyzing telemetry stream packets received by the collector

Intelligent Terminal Products

Product Support Services

Technical Service Solutions

Resource Center

Policy

Online Help

Become a Partner

Partner Policy & Program

Global Learning

Partner Sales Resources

Service Business

News & Events

Contact Us