Troubleshooting network management and monitoring

Troubleshooting NQA

TWAMP Light test failure

Symptom

The device, acting as the TWAMP Light sender, starts a TWAMP Light test to the destination end. The TWAMP Light test fails when one of the following conditions occurs:

·     The status of the TWAMP Light test is abnormal.

Execute the display nqa twamp-light client command on the device. If the Status field value in the output is Inactive, the TWAMP Light test has not started and is considered a failure.

·     The TWAMP Light test result contains an anomaly.

Execute the display nqa twamp-light client statistics two-way-loss test-session command on the device. If the Loss count field value is not 0, it indicates that TWAMP Light test packet loss occurs on the network. If the Error count field value is not 0, it indicates that the device receives error TWAMP Light test packets. When the number of lost packets or error packets exceeds the threshold allowed for user services, the TWAMP Light test is considered a failure.

Common causes

·     The following are the common causes of test status anomaly:

¡     On a Layer 3 VPN network, the VPN is deleted.

¡     On a Layer 2 VPN network, the source AC state changes to down.

¡     An interface card is removed and the interface specified by the source interface command does not exist.

·     The following are the common causes of test result anomaly:

¡     Packet loss occurs.

-     Settings on the TWAMP Light sender and responder do not match.

-     The device cannot communicate with the destination address of the test. The destination address cannot be pinged or packet loss occurs during the ping operation.

-     CRC errors occur on the interface.

¡     Packet error occurs.

-     The timeout time specified by the timeout keyword in the start command is too small. After you execute the start command to start the TWAMP Light test, the reflected package reaches the device after the timeout timer expires. The device takes the reflected packet as an error packet.

-     The content of the test packets contains fields that do not comply with the protocol requirements.

-     Packet encapsulation fails.

Troubleshooting flow

Figure 1 shows the troubleshooting flowchart:

Figure 1 Flowchart for troubleshooting TWAMP Light test failure

 

Solution

1.     Collect the status and results of the failed TWAMP Light test.

Execute the display nqa twamp-light client and display nqa twamp-light client statistics two-way-loss test-session commands on the device. Identify the failed TWAMP Light test and collect its status and results.

¡     Execute the display nqa twamp-light client command on the device. If the Status field value in the output is Inactive, it indicates that the status of the TWAMP Light test is abnormal.

<Sysname> display nqa twamp-light client

Brief information about all test sessions:

Total sessions: 1

Active sessions: 1

-----------------------------------------------------------------------------

ID    Status     Source IP/Port         Destination IP/Port

1     Active     1.1.1.1/10000          1.1.1.2/20000

¡     Execute the display nqa twamp-light client statistics two-way-loss test-session command on the device. If the Loss count field value is not 0, it indicates that TWAMP Light test packet loss has occurred on the network. If the Error count field value is not 0, it indicates that the device receives error TWAMP Light test packets.

<Sysname> display nqa twamp-light client statistics two-way-delay test-session 1

Latest two-way loss statistics:

    Index         Loss count    Loss ratio    Error count    Error ratio

    1             200           100.0000%     0              0.0000%

    2             200           100.0000%     0              0.0000%

    3             200           100.0000%     0              0.0000%

    4             200           100.0000%     0              0.0000%

    5             200           100.0000%     0              0.0000%

--------------------------------------------------------------------------------

Average loss count  : 200             Average loss ratio  : 100.0000%

Maximum loss count  : 200             Maximum loss ratio  : 100.0000%

Minimum loss count  : 200             Minimum loss ratio  : 100.0000%

Average error count : 0               Average error ratio : 0.0000%

Maximum error count : 0               Maximum error ratio : 0.0000%

Minimum error count : 0               Minimum error ratio : 0.0000%

2.     To resolve the test status anomaly, perform the following tasks:

a.     If the device has just completed startup or active/standby switchover, or the interface card where the interface (specified in the source interface command) is located has not completed startup, wait for the device state to become stable. Execute the display system stable state command. If the System state field value in the output is Stable, the device is already in a stable state. In this case, identify whether the test status changes to Active.

-     If it changes to Active, no further action is required.

-     If it does not change to Active, proceed to the next step.

b.     If the device is operating stably, identify whether the configuration is complete.

-     On a Layer 3 VPN network, execute the display nqa twamp-light client verbose command to view the VPN bound to the TWAMP Light test and execute the display ip vpn-instance command to identify whether the VPN exists. If the bound VPN does not exist, execute the ip vpn-instance command in system view to create a VPN instance.

-     On a Layer 2 VPN network, execute the display nqa twamp-light client verbose command to view the source interface. If the Source interface field value is a hyphen (-), execute the source interface command in TWAMP Light client-session view to specify a source AC for test packets, and make sure the specified interface is up.

c.     Identify whether the network connection is ready. If a source interface or source AC is specified for the TWAMP Light test, make sure the source interface or source AC is up.

-     Execute the display l2vpn pw xconnect-group or display l2vpn forwarding ac command. If the State field (which represents AC state) value is Down, resolve the AC issue.

-     Execute the display interface command. If the values for the Current state and Line protocol state fields (which indicate the interface state) are Down, bring up the interface.

3.     To resolve the test packet loss issue, perform the following tasks:

a.     Identify whether packet loss is due to configuration errors.

Execute the display nqa twamp-light client verbose command on the device and the display nqa twamp-light responder command on the TWAMP Light responder of the test. If the following parameters are specified, the settings on the TWAMP Light sender and responder must be consistent.

-     Source IP address. You can edit this parameter on the TWAMP Light sender by using the source ip or source ipv6 command in TWAMP Light client-session view.

-     Source port number. You can edit this parameter on the TWAMP Light sender by using the source port command in TWAMP Light client-session view.

-     Destination IP address. You can edit this parameter on the TWAMP Light sender by using the source ip or source ipv6 command in TWAMP Light client-session view.

-     Destination port number. You can edit this parameter on the TWAMP Light sender by using the destination port command in TWAMP Light client-session view.

-     VPN instance name. You can edit this parameter on the TWAMP Light sender by using the vpn-instance command in TWAMP Light client-session view.

-     VLAN ID. You can edit this parameter on the TWAMP Light sender by using the vlan command in TWAMP Light client-session view.

-     Source MAC address. You can edit this parameter on the TWAMP Light sender by using the source mac command in TWAMP Light client-session view.

-     Destination MAC address. You can edit this parameter on the TWAMP Light sender by using the destination mac command in TWAMP Light client-session view.

You can edit all the above parameters on the TWAMP Light responder by using the test-session command in TWAMP Light responder view.

If the timestamp format specified on the TWAMP Light sender is NTP and the test packet sending interval is 10 or 100 milliseconds, the device takes it as a configuration conflict. In this case, the TWAMP Light test fails. Execute the start command in TWAMP Light sender view to edit the test packet sending interval, or execute the timestamp-format command in TWAMP Light client-session view to edit the timestamp format.

b.     Execute the terminal monitor, terminal debugging, debugging nqa error, and debugging nqa event commands sequentially in user view on the TWAMP Light sender to enable debugging for NQA. Then, execute the view /var/log/trace.log command in probe view to view NQA trace logs. Based on the logs, identify whether the device correctly sends TWAMP Light test packets and receives reflected packets of the TWAMP Light test, and whether the timestamps in the test results are correct.

-     If the TWAMP Light sender does not send test packets correctly, identify the cause of the packet sending issue based on the NQA debugging information and trace logs. Edit the TWAMP Light test configuration on the sender according to the failure reason, and then restart the TWAMP Light test. If the packet sending issue cannot be resolved, execute the following commands, collect command output information, and then proceed to step 5.

display ip statistics

display ipv6 statistics

display ethernet statistics

-     If the TWAMP Light sender does not receive reflected packets of the test correctly, execute the nqa agent enable command in system view of the TWAMP Light responder to enable the NQA client. Then, return to user view and execute the terminal monitor, terminal debugging, and debugging nqa packet commands sequentially to enable NQA packet debugging. Identify whether the TWAMP Light responder receives NQA packets and whether the NQA packet configuration is correct. If the responder does not receive any NQA packet, the network might fail. Proceed to the next step to troubleshoot the network issue. If the NQA packet configuration is incorrect, edit the NQA configuration as described in step 3.a and restart the test. The following example illustrates the NQA packet debugging information:

-     Identify whether the following timestamp conditions are met in the test results:

CSendTime ≤ CRecvTime.

SRecvTime ≤ SSendTime.

SSendTime – SRecvTime, which is the processing time of the NQA server, is relatively small.

If these conditions are not met, collect the timestamp values and execute the display device command to collect device card information, and then proceed to step 5.

The following examples illustrate trace logs:

*May  6 00:36:24:900 2023 Sysname NQA/7/KDIAG: send packt, session 1, ucSampler 187.

// The output shows that the device sends a TWAMP Light test packet.

*May  6 00:36:24:901 2023 Sysname NQA/7/KDIAG: Twmap Recv Pakcet ucSampler=187

// The output shows that the device receives a TWAMP Light reflected packet.

*May  6 00:36:24:901 2023 Sysname NQA/7/KDIAG: cSendSec is 1683304584, cSendFrac is 900923500, sRecvSec is 1683304584, sRecvFrac is 835000000,cRecvSec is 1683304584, cRecvFrac is 901923500, sSendSec is 1683304584, sSendFrac is 835000000

*May  6 00:36:24:901 2023 Sysname NQA/7/KDIAG: nqa entry (twamplight?session-1) Sampler(187) client time:

  CSendTime=1683304584900923    CRecvTime=1683304584901923      SRecvTime=1683304584835000      SSendTime=1683304584835000

// The output shows the timestamps within a TWAMP Light test packet.

c.     Identify whether packet loss is caused by network failures. Execute the ping command to ping the destination address of the test. If the destination address cannot be pinged or packet loss occurs, first resolve the network failures.

d.     Identify whether the packet loss is caused by CRC errors.

e.     Execute the display counters command. If the value of the Err (pkts) field in the command output keeps increasing as the test progresses, it indicates that packet sending failures occur on the link layer. Replace the associated interfaces or cables to resolve the issue.

4.     To resolve the error packet issue, perform the following tasks:

a.     Identify whether configuration errors exist and the device mistakenly considers TWAMP Light reflected packets that arrive late as error packets.

-     Execute the ping command on the TWAMP Light sender to ping the responder and view the maximum delay between the two ends, which is the max value in the round-trip min/avg/max/std-dev field of the ping results, in milliseconds.

-     Execute the display nqa twamp-light client verbose command on the device and view the value of the Timeout(sec) field, which represents the timeout time of the TWAMP Light reflected packets. The timeout time of TWAMP Light reflected packets must be greater than the maximum delay between the two ends. If it is not, execute the start command in TWAMP Light sender view to specify a larger timeout time with the time-out keyword.

b.     Execute the terminal monitor, terminal debugging, debugging nqa error, and debugging nqa event commands sequentially in user view on the TWAMP Light sender to enable debugging for NQA. Then, execute the view /var/log/trace.log command in probe view to view NQA trace logs. Based on the logs, identify whether the packet content meets protocol requirements and whether the packet encapsulation is correct. If the packet content does not meet protocol requirements or the packet encapsulation is incorrect, re-configure the TWAMP Light test as described in TWAMP Light configuration.

5.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module name: HH3C-TWAMP-MIB

·     hh3cTwampTwoWayLossExceed (1.3.6.1.4.1.25506.2.184.1.0.1)

·     hh3cTwampTwoWayLossRecover (1.3.6.1.4.1.25506.2.184.1.0.2)

·     hh3cTwampTwoWayDelayExceed (1.3.6.1.4.1.25506.2.184.1.0.3)

·     hh3cTwampTwoWayDelayRecover (1.3.6.1.4.1.25506.2.184.1.0.4)

·     hh3cTwampTwoWayJitterExceed (1.3.6.1.4.1.25506.2.184.1.0.5)

·     hh3cTwampTwoWayJitterRecover (1.3.6.1.4.1.25506.2.184.1.0.6)

·     hh3cTwampSenderStartFailure (1.3.6.1.4.1.25506.2.184.1.0.9)

·     hh3cTwampStatisticsAbnormal (1.3.6.1.4.1.25506.2.184.1.0.11)

Log messages

·     NQA/6/NQA_TWAMP_LIGHT_PACKET_INVALID

·     NQA/6/NQA_TWAMP_LIGHT_REACTION

·     NQA/6/NQA_TWAMP_LIGHT_SENDER_START_FAILURE

·     NQAS/6/NQA_TWAMP_LIGHT_START_FAILURE