Login
- Released At: 30-04-2024
- Page Views:
- Downloads:
- Table of Contents
- Related Documents
-
|
|
|
H3C SecPath Security Products |
|
Comware 7 Troubleshooting Guide |
|
|
|
|
Document version: 6W100-20230413
Copyright © 2023 New H3C Technologies Co., Ltd. All rights reserved.
No part of this manual may be reproduced or transmitted in any form or by any means without prior written consent of New H3C Technologies Co., Ltd.
Except for the trademarks of New H3C Technologies Co., Ltd., any trademarks that may be mentioned in this document are the property of their respective owners.
The information in this document is subject to change without notice.
Contents
Applicable device models and software versions
Collecting log and operating information
Collecting common log messages
Collecting diagnostic log messages
Collecting operating statistics
Failure model and impact analysis
Common service recovery and fault removal methods
Abnormal card state or card failure
Troubleshooting packet forwarding failures
Device ping failure from a directly connected PC
Connectivity failure between two PCs connected through the device
Connectivity failure between PCs connected through the device in the same security zone
Ping or tracert operation failure
Ping operation failure across NAT
IRF fabric establishment failure
Troubleshooting RBM dynamic routing issues
RBM switchover not triggered upon uplink or downlink interface failure
Inconsistent ACL configuration between the RBM member devices
A Reth interface is not pingable if not assigned to redundancy group
Failure to access the external network from internal users
Source address translation failure
Destination address translation failure
IPsec configuration failure (NAT in unification with IPsec)
Failure to access the gateway device configured with policy-based NAT from internal users
Failure to access the gateway device configured with source address translation from external users
Failure to access the external network from internal users
Source address translation failure
Destination address translation failure
IPsec configuration failure (NAT in unification with IPsec)
Failure to access the gateway device configured with source address translation from external users
NAT fails but the outbound interface can be pinged successfully from the external network
IPv6 access to IPv4 network fails
IPsec SAs established successfully but IPsec-protected traffic cannot be forwarded
IPsec exceptions occur when the master device in IRF fabric goes down
IKE SAs established successfully but IPsec SAs cannot be established
IPsec smart link does not detect link quality
IPsec tunnel interface-based IPsec tunnel cannot be established
Troubleshooting load balancing
High CPU usage and memory usage
Troubleshooting system management
Failure to log in to the SSL VPN Web interface
Failure to log in to the SSL VPN gateway from a browser
Failure to access internal resources from a browser
Failure to obtain SSL VPN gateway information from an iNode client
Failure to log in to the SSL VPN gateway from an iNode client
Failure to access internal resources from an iNode client
Failure to terminate idle SSL VPN sessions of iNode client users
User filtering, monitoring, and IP binding settings not take effect
Failure to relog in to the SSL VPN gateway
Failure to configure WeChat Work (or WeCom) authentication
IPS or anti-virus mistakenly intercepts legal traffic
IPS or WAF fails to intercept attack traffic or generate attack logs
Application rate limit does not take effect
File filtering or data filtering does not take effect
SSL decryption does not take effect
Application audit and management does not take effect
URL filtering does not take effect
Server connection detection does not take effect
IP reputation does not take effect
Data analysis center fails to display or refresh logs
Troubleshooting high CPU usage caused by policy rule matching acceleration
CPU usage is high if object policy rules are modified frequently
High CPU usage caused by low-speed security policy matching
Troubleshooting attack detection and prevention failures
FIN flood attack report failure
Troubleshooting threat log generation by IPS
No threat logs generated on the IPS device
Troubleshooting RBM dynamic routing issues
RBM switchover not triggered upon uplink or downlink interface failure
Inconsistent ACL configuration between the RBM member devices
IPv6 access to IPv4 network fails
Unexpected card reboot because of an internal port failure
Unexpected power-off of a card because of an internal port failure
Electronic label reading failure
MPU and service module version inconsistency
Troubleshooting load balancing
Health monitoring failure with a TCP probe template
Health monitoring failure with an HTTP probe template
Health monitoring failure with a UDP probe template
Failed to import a CA certificate to a PKI domain due to lack of a key file
Inactive state of a virtual server
HTTP service access failure for a TCP virtual server
XFF failure for an HTTP virtual server
Domain name resolution failure for global load balancing
Outbound link load balancing failure
Applicable device models and software versions
This document is not restricted to specific software or hardware versions. Procedures and information in the document might be slightly different depending on the software or hardware version of the device.
Introduction
This document provides information about troubleshooting common software and hardware issues with the device.
General guidelines
|
|
IMPORTANT: To prevent an issue from causing loss of configuration, save the configuration each time you finish configuring a feature when the system is operating correctly. For configuration recovery, regularly back up the configuration to a remove server. |
When you troubleshoot the device, follow these general guidelines:
· To ensure safety, wear an ESD wrist strap when you replace or maintain a hardware component.
· Device failures include MPU failures, service module failures, interface module failures, and switching fabric module failures. You can collect information about MPU and interface module failures from the console port on the MPU or through Telnet. Information about service module failures can be mainly collected from the console port on the service module. (Applicable to distributed devices.)
· To help identify the cause of the issue, collect system and configuration information, including:
¡ Symptom, time of failure, and configuration.
¡ Network topology information, including the network diagram, port connections, and points of failure.
¡ Steps you have taken, such as reconfiguration, cable swapping, and reboot, and the action results.
¡ Output from the commands executed during the troubleshooting process.
¡ Log messages and diagnostic information.
¡ Information about the captured packets, debugging information, information about repeated reboots of an MPU or switching fabric module.
¡ Physical evidence of failure:
- Photos of the hardware.
- Status of the card, power, and fan tray status LEDs.
· When a service module failure occurs, record failure information about the service module separately. Collect a console cable to the console port on the service module to collect related information.
· Be clear about the impact of each configuration or operation during the troubleshooting process and make sure any issues caused by the configuration or operation can be fixed and will not cause serious consequences.
· After each operation is performed, wait for a certain time to verify the operation effect.
· To prevent configuration loss, do not save the configuration in the troubleshooting process especially when an IRF split occurs.
Collecting log and operating information
|
|
IMPORTANT: By default, the information center is enabled. If the feature is disabled, you must use the info-center enable command to enable the feature for collecting log messages. |
The device generates common and diagnostic log messages and operating statistics during the operating process. This information is stored in flash. You can export this information by using FTP or TFTP. To more easily locate log and operating information, use a consistent rule to categorize and name files exported from different MPUs. For example, save log information files to a separate folder for each MPU, and include their chassis and slot numbers in the folder names.
The common log messages are in the log buffer before they are saved to the log file. The system saves the contents in the log file buffer to the log file at the specified frequency. You can also execute the logfile save command in any view to save the contents of the log file buffer to the log file immediately.
The diagnostic log messages are in the diagnostic log buffer before they are saved to a diagnostic log file. The system saves the contents of the diagnostic log file buffer to the diagnostic log file at the specified frequency. You can also execute the diagnostic-logfile save command in any view to save the contents of the diagnostic log file buffer to the diagnostic log file immediately.
These log files are stored in flash or CF card of the device. You can export these files by using FTP or TFTP.
Table 1 Log and operating information
|
Category |
File name format |
Content |
|
Common log |
logfileX.log |
Command execution, trap, and operational log messages. |
|
Diagnostic log |
diagfileX.log |
Diagnostic log messages about device operation, including the following items: · Parameter settings in effect when an error occurs. · Information about a card startup error. · Handshaking information between the MPU and interface card when a communication error occurs. |
|
Operating statistics |
file-basename.gz |
Device status, CPU status, memory status, configuration, software entries, and hardware entries |
Collecting common log messages
1. Use the logfile save command to save common log messages from the log buffer to the log file. The common log messages include active MPU log messages, standby MPU log messages, and active MPU log message on each IRF member device, and standby MPU log message on each IRF member device. If contexts have been created on the device, you need also to collect log messages of each context.
[H3C] logfile save
The contents in the log file buffer have been saved to the file flash:/logfile/l
ogfile.log.
2. Verify information about the log file.
<sysname> dir flash:/logfile/
Directory of flash:/logfile
0 -rw- 10483632 Jul 08 2014 15:05:22 logfile.log
253156 KB total (77596 KB free)
Collecting diagnostic log messages
1. Use the diagnostic-logfile save command to save diagnostic log messages from the diagnostic log buffer to the diagnostic log file. The more modules on the device, the longer the system takes to collect diagnostic log messages. During the diagnostic log message collection process, do not execute any command. Be patient and wait until the collection completes.
<sysname>diagnostic-logfile save
The contents in the diagnostic log file buffer have been saved to the file flash
:/diagfile/diagfile.log.
2. Verify information about the diagnostic log file.
<sysname>dir flash:/diagfile/
Directory of flash:/diagfile
0 -rw- 10485740 Nov 04 2020 17:51:52 diagfile.log
7456492 KB total (6624504 KB free)
Collecting operating statistics
You can use the display diagnostic-information command to display or save diagnostic information.
# To save operating statistics, execute the display diagnostic-information command and enter Y to save diagnostic information when you are promoted to save or display diagnostic information. The information might fail to be collected completely if you enter N to display diagnostic information.
<sysname> display diagnostic-information
Save or display diagnostic information (Y=save, N=display)? [Y/N]:y
Please input the file name(*.gz)[flash:/diag.gz]:flash:/diag.gz
Diagnostic information is outputting to flash:/diag.gz.
Save successfully.
<sysname> dir flash:/
Directory of flash:
......
6 -rw- 898180 Jun 26 2013 09:23:51 diag.gz
1021808 KB total (259072 KB free)
# To display diagnostic information, execute the screen-length disable command to disable pausing between screens of output before executing the display diagnostic-information command.
<sysname> screen-length disable
% Screen-length configuration is disabled for current user.
<sysname> display diagnostic-information
Save or display diagnostic information (Y=save, N=display)? [Y/N]:N
==================================================================
===============display cpu===============
Slot 1 CPU 0 CPU usage:
6% in last 5 seconds
6% in last 1 minute
6% in last 5 minutes
===========================================================
=================================================================
===============display cpu-usage history slot 1 ===============
100%|
95%|
90%|
85%|
80%|
75%|
70%|
65%|
60%|
55%|
50%|
45%|
40%|
35%|
30%|
25%|
20%|
15%|
10%|
5%|############################################################
------------------------------------------------------------
10 20 30 40 50 60 (minutes)
cpu-usage (Slot 1 CPU 0) last 60 minutes (SYSTEM)
……………………………………
Troubleshooting methods
When a fault occurs, first collect related system, configuration, and operation information to identify the fault type and then resolve the fault according to the troubleshooting method for this fault type.
If the fault cannot be identified, provide the fault description together with collected information to the technical support for analysis.
Troubleshooting flowchart
Figure 1 shows the general troubleshooting process for you to identify the fault type.
Figure 1 Troubleshooting flowchart
Fault location methods
The common fault location methods include:
· View packet statistics on interfaces.
· Mirror traffic on interfaces.
· Capture packets on interfaces.
· View session status and statistics.
· View Layer 2 and Layer 3 forwarding entries and statistics.
· Identify whether OpenFlow entries are deployed correctly.
Fault types
The following types of faults might occur on the device:
· Chassis failure—Unexpected reboot, abnormal state, failure to start up, or repeated reboots. For the troubleshooting procedure, see "Chassis failure" in "Troubleshooting hardware."
· Module failure—Unexpected reboot, abnormal state, failure to start up, or repeated reboots. For the troubleshooting procedure, see "Abnormal card state or card failure" in "Troubleshooting hardware."
· Temperature alarm—For the troubleshooting procedure, see "Temperature alarms" in "Troubleshooting hardware."
· Interface failure—If an interface fails to come up, flaps between up and down states, or has error packets, see "Troubleshooting interfaces" for the troubleshooting procedure.
· IRF failure—If devices fail to form an IRF or an IRF split occurs, see "Troubleshooting IRF" for the troubleshooting procedure.
· Hot backup failure—If an exception occurs during master/subordination switchover, forwarding through the redundant port, or service switching to a redundant port, see "Troubleshooting hot backup" for the troubleshooting procedure.
· Load balancing failure—See "Troubleshooting load balancing"
· " for the troubleshooting procedure.
· NAT/ALG failure—See "Troubleshooting policy NAT", "Troubleshooting interface NAT", and "Troubleshooting AFT" for the troubleshooting procedure.
· IPsec/IKE failure—See "Troubleshooting IPsec and IKE" for the troubleshooting procedure.
· High CPU usage—See "High CPU usage in "Troubleshooting system management" for the troubleshooting procedure.
· High memory usage—See "High memory usage" in "Troubleshooting system management" for the troubleshooting procedure.
Failure model and impact analysis
Figure 2 shows a typical network failure model. In this model, the two devices set up an IRF HA system and work in dual-active or active-standby mode for network availability.
Figure 2 Network failure model
Table 2 Failure point impact analysis
|
Failure point |
Possible symptoms |
Impact |
|
(1) and (3), having transceiver modules |
Interface down |
Service switchover. |
|
Increase of error packets on the interface |
Services on the link will be affected (wide-reaching impact). |
|
|
(2) |
MPU failure |
Service switchover. |
|
Service module failure |
If the link tracks the service module status, service switchover will occur. |
|
|
Interface module failure |
Service switchover might occur. |
|
|
(4) |
Single IRF link failure |
Services are unaffected, but performance might be degraded. |
|
Dual IRF link failure |
IRF split. |
Common service recovery and fault removal methods
Table 3 Common power and air conditioning facilities methods
|
Fault category |
Service recovery methods |
Fault removal methods |
|
Hardware |
· Isolate the faulty card. · Isolate the faulty device by adjusting the traffic forwarding paths. For example, changing the preferences of routes so traffic is switched to other paths. |
Complete required tests on the backup hardware, and replace the failed hardware. |
|
Software |
· Re-enable the protocols on the faulty device. · Isolate the faulty device by adjusting the traffic forwarding paths. |
Upgrade the software version, including the patch version. Adjust the network topology or modify the configuration to remove the failures |
|
Link |
Isolate the faulty link by adjusting the traffic forwarding paths. |
Remove link errors. |
|
Others |
· Correct configuration errors. · Connect the ports of the devices correctly. · Isolate the faulty link by adjusting the traffic forwarding paths. |
· Modify the incorrect configurations. · Correctly connect the device ports. · Repair the power and air conditioner systems for the devices. |
Troubleshooting hardware
Chassis failure
Symptom
The chassis reboots unexpectedly.
Solution
To resolve the issue:
Execute the display version command, and check the last reboot reason field to identify the reason for the chassis reboot.
If the chassis reboot is caused by software anomalies, collect diagnostic information and send it to the H3C Support.
H3C Comware Software, Version 7.1.064, Ess 8601P08
Copyright (c) 2004-2019 New H3C Technologies Co., Ltd. All rights reserved.
H3C SecPath F1090 uptime is 0 weeks, 0 days, 0 hours, 5 minutes
Last reboot reason: User reboot
Boot image: flash:/F1090FW-CMW710-BOOT-E8601P08.bin
Boot image version: 7.1.064, Ess 8601P08
Compiled Sep 10 2019 15:00:00
System image: flash:/F1090FW-CMW710-SYSTEM-E8601P08.bin
System image version: 7.1.064, Ess 8601P08
Compiled Sep 10 2019 15:00:00
SLOT 1
CPU type: Multi-core CPU
DDR4 SDRAM Memory: 8192M bytes
FLASH: 7296M bytes
CPLD_A Version: 1.0
CPLD_B Version: 1.0
Release Version:SecPath F1090-8601P08
Basic BootWare Version:0.30
Extend BootWare Version:1.01
BuckleBoard Version:Ver.A
BackBoard1 Version:Ver.A
BackBoard2 Version:Ver.A
HD_BackBoard Version:Ver.D
Pcb Version:Ver.A
[SUBCARD 0] NSQ1F1MSPUOTXA(Hardware)Ver.A, (Driver)1.0, (Cpld)1.0
Boot Type: Warm
[H3C]display system internal version
H3C SecPath F1090 V800R006B01D645SP08
Comware V700R001B64D045SP08
Abnormal card state or card failure
Unless otherwise stated, MPU, interface modules (or LPUs), service modules (for example, firewall modules), switching fabric modules (or NPUs), and other functional hardware modules are collectively referred to as cards in this document.
The card states include Normal, Master, Standby, Absent, and Fault.
· Normal—The card is operating correctly.
· Master—The card is the active MPU.
· Standby—The card is the standby MPU.
· Absent—The card is absent.
· Fault—The card is faulty.
Symptom
The output from the display device command shows that a card is in Absent or Fault state.
<sysname>display device
Slot No. Brd Type Brd Status Subslot Sft Ver Patch Ver
0 NSQM1CGQ4TG24SHA0Normal 0 M9016-V-9153P22 None
1 NONE Absent 0 NONE None
2 NSQM1CGQ4TG24SHA0Normal 0 M9016-V-9153P22 None
3 NONE Absent 0 NONE None
4 NSQM1SUPD0 Master 0 M9016-V-9153P22 None
5 NSQM1SUPD0 Standby 0 M9016-V-9153P22 None
6 NSQM1FWEFGA0 Normal 0 M9016-V-9153P22 None
CPU 1 Normal 0 M9016-V-9153P22
7 NONE Absent 0 NONE None
8 NONE Absent 0 NONE None
9 NONE Absent 0 NONE None
10 NSQM1FAB08E0 Normal 0 M9016-V-9153P22 None
11 NSQM1FAB08E0 Normal 0 M9016-V-9153P22 None
12 NSQM1FAB08E0 Normal 0 M9016-V-9153P22 None
13 NSQM1FAB08E0 Normal 0 M9016-V-9153P22 None
Solution
Handling a card in Absent state
To resolve the issue:
1. Verify that the card is installed securely. Reinstall the card to ensure that the card is installed securely.
2. Verify that the card is not faulty.
a. Install this card into another slot.
b. Install another card that is operating correctly on the chassis into this slot.
3. Verify that the LEDs of card do not indicate any error.
4. If the card is an MPU, service module, or switching fabric module with a console port, connect the card to a configuration terminal to verify that it can start up correctly.
5. If the card is faulty, collect fault information, replace the card, and contact H3C Support.
Handling a card in Fault state
To resolve the issue:
1. Wait approximately 10 minutes, and then check the card status:
¡ If the card remains in Fault state, go to the next step.
¡ If the card state changes to Normal, and then reboots, contact H3C Support.
2. If the card is an MPU or switching fabric module with a console port, connect the card to a configuration terminal through a console cable and verify that the module can start up correctly.
3. Install the card into another slot to determine whether the card is faulty.
4. If the card is faulty, collect fault information, replace the card, and contact H3C Support.
Fan tray failure
Symptom
The fan tray status LED indicates an abnormal condition exists. The device outputs messages about fan tray failures as follows:
%May 06 10:12:24:805 2017 H3C DEV/3/FAN_ABSENT: -MDC=1; Slot 2 Fan 2 is absent.
%May 06 10:12:32:805 2017 H3C DEVD/2/DRV_DEV_FAN_CHANGE: -MDC=1; Slot 2: Fan communication state changed: Fan 1 changed to fault.
%May 06 10:12:42:405 2017 H3C DEV/2/FAN_FAILED: -MDC=1; Slot 2 Fan 1 failed.
Solution
To resolve the issue:
1. If the fan tray is present in the slot, place your hand at the outlet air vents of the device to verify that wind blows out of the device.
If no wind blows out of the device, the fan tray is faulty.
2. Verify that the inlet and outlet air vents are not blocked and no large amount of dust buildup exists on the inlet and outlet air vents.
3. Verify that the fan tray is present in the slot with normal operating state and normal fan speed.
Execute the display fan command to view the fan tray operating status information. If fan status is not normal, or the displayed fan speed is less than half of the normal fan speed, you can remove and reinstall the fan tray or swap the fan tray with another to verify the failure reason.
<sysname> display fan
SLOT 1 Fan 0 Status: Normal Speed:9500
SLOT 1 Fan 1 Status: Normal Speed:9500
SLOT 1 Fan 2 Status: Normal Speed:9500
4. If the issue persists, replace the fan tray.
If no fan tray is present, power off the device to avoid module damage caused by high temperature. You can continue using the device if you can use cooling measures to keep the device operating temperature below 50°C (122°F).
5. If the issue persists, contact H3C Support.
Temperature alarms
Symptom
The device outputs a high-temperature or low-temperature alarm message as follows:
%Mar 18 04:22:05:893 2017 H3C DEV/4/TEMPERATURE_WARNING: -Context=1; Temperature is greater than the high-temperature warning threshold on slot 2 sensor inflow 1. Current temperature is 43 degrees centigrade.
Solution
To resolve the issue:
1. Verify that the environment temperature is normal.
If the environment temperature is high, verify the cause of high temperature, such as poor ventilation in the equipment room or failure of the air conditioner.
2. Verify that the device temperature does not exceed the upper or lower warning or alarm thresholds.
You can execute the display environment command to view the module temperature or use hands to touch the modules. If the module temperature is high, immediately examine the causes of high temperature to avoid module damage caused by long-time high temperature of the module.
If the Temperature field displays error or a value out of the ordinary, the switch might fail to access the card temperature sensor through the I2C bus. The switch accesses the transceiver modules through the same I2C bus. You can view whether the transceiver module information is displayed correctly. If the switch can access the transceiver modules, use the temperature-limit command to reconfigure the temperature thresholds. Then use the display environment command to view whether the setting takes effect.
[H3C] temperature-limit slot 1 inflow 1 -5 43 51
[H3C] display environment
System Temperature information (degree centigrade):
--------------------------------------------------------------------------------
---------
Slot Sensor Temperature LowerLimit Warning-UpperLimit Alarm-UpperLimit S
hutdown-UpperLimit
1 inflow 1 29 -5 43 51
NA
2 inflow 1 28 -5 48 56
NA
3. If you still cannot find the cause of temperature alarms, obtain the temperature alarm logs and temperature information and send them to H3C Support for help.
Related commands
|
Command |
Description |
|
display device |
Displays device information. |
|
display environment |
Displays temperature information. |
|
display fan |
Displays the built-in fan tray status. |
|
display power |
Displays power supply information, including: · Power supply status. · Power supply type, rated input voltage, and rated output voltage. · Number of available power supplies, total available power of power supplies, total used power, and redundant power. · Status of installed power supplies. · Power supply status of the cards. |
|
display version |
Displays system version information, module uptime, and last reboot reason. |
|
save |
Saves the running configuration to a configuration file. |
|
temperature-limit |
Sets the temperature alarm thresholds. |
Troubleshooting interfaces
Error packets on an interface
Symptom
The output from the display interface command shows that error packets exist on an interface.
<sysname>display interface GigabitEthernet 1/0/2
GigabitEthernet1/0/2
Current state: DOWN
Line protocol state: DOWN
Description: GigabitEthernet1/0/2 Interface
Maximum transmission unit: 1500
Internet address: 192.168.2.1/24 (primary)
IP packet frame type: Ethernet II, hardware address: 50da-00dd-1327
IPv6 packet frame type: Ethernet II, hardware address: 50da-00dd-1327
Media type is twisted pair, loopback not set, promiscuous mode not set
Speed Negotiation, Duplex Negotiation, link type is autonegotiation
Output flow-control is disabled, input flow-control is disabled
Last link flapping: Never
Last clearing of counters: Never
Peak input rate: 0 bytes/sec, at 00-00-00 00:00:00
Peak output rate: 0 bytes/sec, at 00-00-00 00:00:00
Last 300 second input: 0 packets/sec 0 bytes/sec -%
Last 300 second output: 0 packets/sec 0 bytes/sec -%
Input (total): 0 packets, 0 bytes
0 unicasts, 0 broadcasts, 0 multicasts, 0 pauses
Input (normal): 0 packets, 0 bytes
0 unicasts, 0 broadcasts, 0 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, - throttles
0 CRC, 0 frame, 0 overruns, 0 aborts
0 ignored, - parity errors
Output (total): 0 packets, 0 bytes
0 unicasts, 0 broadcasts, 0 multicasts, 0 pauses
Output (normal): 0 packets, 0 bytes
0 unicasts, 0 broadcasts, 0 multicasts, 0 pauses
Output: 0 output errors, 0 underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, 0 no carrier
Fields for incoming error packets
· input errors—Total number of incoming error packets.
· runts—Number of incoming frames that meet the following conditions:
¡ Shorter than 64 bytes.
¡ In correct format.
¡ Containing valid CRCs.
· giants—Number of incoming giants. Giants refer to frames larger than the maximum frame length supported on the interface. For an Ethernet interface that does not permit jumbo frames, the maximum frame length is 1518 bytes (without VLAN tags) or 1522 bytes (with VLAN tags). For an Ethernet interface that permits jumbo frames, the maximum Ethernet frame length is set when you configure jumbo frame support on the interface.
· throttles—Number of incoming frames that had a non-integer number of bytes.
· CRC—Total number of incoming frames that had a normal length but contained CRC errors.
· frame—Total number of incoming frames that contained CRC errors and a non-integer number of bytes.
· overruns—Number of packets dropped because the input rate of the port exceeded the queuing capability.
· aborts—Total number of illegal incoming packets, including fragment frames, jabber frames, symbol error frames, unknown operation code frames, and length error frames.
· ignored—Number of incoming frames dropped because the receiving buffer of the port ran low.
· parity errors—Total number of frames with parity errors.
Fields for outgoing error packets
· output errors—Total number of outgoing packets with errors.
· underruns—Number of packets dropped because the output rate of the interface exceeded the output queuing capability. This is a low-probability hardware anomaly.
· buffer failures—Number of packets dropped because the transmitting buffer of the interface ran low.
· aborts—Number of packets that failed to be transmitted. Transmission of these packets had started, but failed because of various reasons (for example, collision).
· deferred—Number of frames that the interface deferred to transmit because of detected collisions.
· collisions—Number of frames that the interface stopped transmitting because collisions were detected during transmission.
· late collisions—Number of frames that the interface deferred to transmit after transmitting their first 512 bits because of detected collisions.
· lost carrier—Number of carrier losses during transmission. This counter increases by one when a carrier is lost, and applies to serial WAN interfaces.
· no carrier—Number of times that the port failed to detect the carrier when attempting to send frames. This counter increases by one when a port failed to detect the carrier, and applies to serial WAN interfaces.
Solution
To resolve the issue, choose one of the following solutions depending on the symptom:
· Solution for increasing CRC, frame, and throttles errors in the inbound direction
· Solution for increasing giants in the inbound direction
· Solution for increasing error packets in the outbound direction
Solution for increasing CRC, frame, and throttles errors in the inbound direction
1. Test the link performance. If the link is of poor quality or optical signals are attenuated greatly, replace the cable or optical fiber.
2. If the interface is installed with a transceiver module, identify whether the issue is caused by a transceiver module failure as described in "Transceiver module failure."
3. Swap the cable, optical fiber, or transceiver module with that of an interface that is operating correctly, and then swap it over.
¡ If the issue remains the same on the original interface but does not occur on the new interface, the original interface might be the failure cause. Use an interface that can operate correctly to provide services, and send the failure information to H3C Support for analysis.
¡ If the issue does not occur on the original interface but occurs on the new interface, verify that the peer device and the intermediate devices and links are operating correctly.
4. If the issue persists, contact H3C Support.
Solution for increasing giants in the inbound direction
1. Examine the following settings of the jumboframe enable command for the interfaces on two ends:
¡ Verify that the jumbo feature is enabled on both interfaces.
¡ Verify that the default settings for the command are the same.
¡ Verify that the current settings for the command are the same.
2. If the issue persists, contact H3C Support.
Solution for increasing error packets in the outbound direction
1. Verify that the interface is operating in full duplex mode.
2. If the issue persists, contact H3C Support.
Interface fails to come up
Symptom
An interface fails to come up.
Solution
To resolve the issue:
1. Verify that the cables or optical fibers connected to the interface and its peer interface are connected correctly and securely.
2. If the issue persists, swap the cables or optical fibers for cables or optical fibers that can correctly operate to verify that the intermediate link is operating correctly.
3. Examine the settings of the interfaces, including up/down state, duplex mode, speed, autonegotiation mode, and MDI. Verify that the interfaces are configured correctly.
4. If the interfaces are installed with transceiver modules, verify that the transceiver modules are the same type (including the speed, wavelength, single-mode, and multiple-mode).
5. If the issue persists, swap the suspected transceiver module for a transceiver module that can operate correctly. Identify whether the issue is caused by a transceiver module failure as described in "Transceiver module failure."
<sysname> display transceiver interface GigabitEthernet 1/0/17
GigabitEthernet1/0/17 transceiver information:
Transceiver Type : 1000_BASE_SX_SFP
Connector Type : LC
Wavelength(nm) : 850
Transfer Distance(m) : 550(OM2),270(OM1)
Digital Diagnostic Monitoring : YES
Vendor Name : JDSU
6. If a transceiver module failed, replace the transceiver module and contact H3C Support.
An interface goes down
Symptom
An interface goes down.
Solution
To resolve the issue:
1. Read the log messages for the local and peer devices. Identify whether the interfaces were manually shut down.
2. Display interface status information. Identify whether an interface has protocol issues or was shut down by the diagnostic module because of errors. If yes, contact H3C Support.
<sysname> display interface GigabitEthernet 1/0/2
GigabitEthernet1/0/2
Current state: DOWN
Line protocol state: DOWN
Description: GigabitEthernet1/4/0/1 Interface
Bandwidth: 1000000kbps
Maximum Transmit Unit: 1500
Internet protocol processing: disabled
IP Packet Frame Type:PKTFMT_ETHNT_2, Hardware Address: 8042-0004-5601
IPv6 Packet Frame Type:PKTFMT_ETHNT_2, Hardware Address: 8042-0004-5601
Media type is not sure,Port hardware type is No connector
Last clearing of counters: 16:45:01 Wed 12/11/2013
Peak value of input: 0 bytes/sec, at 2013-12-11 16:45:03
Peak value of output: 0 bytes/sec, at 2013-12-11 16:45:03
Last 300 second input: 0 packets/sec 0 bytes/sec
Last 300 second output: 0 packets/sec 0 bytes/sec
3. As described in Interface fails to come up, verify that the interfaces are correctly configured and the cable, transceiver module, and optical fiber are operating correctly.
4. If the issue persists, contact H3C Support.
Interface state flapping
Symptom
An interface flaps between the up and down states.
Solution
To resolve the issue:
1. If the interface is a fiber port, verify that the transceiver modules at the two ends are operating correctly as described in "Transceiver module failure."
2. If the interface is a copper port, set the speed and duplex mode. The state flapping issue typically occurs in autonegotiation mode. Disable the autonegotiation mode, and configure the same speed and duplex mode for both of the interfaces on two ends.
3. Check the link, peer device, and intermediate devices.
4. If the issue persists, contact H3C Support.
Transceiver module failure
Symptom
A fiber port installed with a transceiver module cannot operate correctly.
Solution
To resolve the issue:
1. If the interface is a 10-GE fiber port, identify whether the fiber port is installed with a Gigabit transceiver module, which is not supported by the fiber port. If yes, replace the transceiver module with one of the supported model.
2. Execute the display transceiver alarm interface command to examine the alarms present on the transceiver module.
¡ If input errors occurred, verify that the peer port, fiber, and intermediate device are operating correctly.
¡ If output errors, current errors, or voltage errors occurred, verify that the local port is operating correctly.
<sysname> display transceiver alarm interface Ten-GigabitEthernet 1/0/25
Ten-GigabitEthernet1/0/25 transceiver current alarm information:
RX signal loss
Table 4 Transceiver module alarms
|
Field |
Description |
|
SFP/SFP+ |
|
|
RX loss of signal |
Incoming (Rx) signal is lost. |
|
RX power high |
Incoming (Rx) power is high. |
|
RX power low |
Incoming (Rx) power is low. |
|
TX fault |
Transmit fault. |
|
TX bias high |
Tx bias current is high. |
|
TX bias low |
Tx bias current is low. |
|
TX power high |
Tx power is high. |
|
TX power low |
Tx power is low. |
|
Temp high |
Temperature is high. |
|
Temp low |
Temperature is low. |
|
Voltage high |
Voltage is high. |
|
Voltage low |
Voltage is low. |
|
Transceiver info I/O error |
Transceiver information read and write error. |
|
Transceiver info checksum error |
Transceiver information checksum error. |
|
Transceiver type and port configuration mismatch |
The transceiver type does not the match port configuration. |
|
Transceiver type not supported by port hardware |
The port does not support the transceiver type. |
|
XFP |
|
|
RX loss of signal |
Incoming (Rx) signal is lost. |
|
RX not ready |
The receiver is not ready. |
|
RX CDR loss of lock |
Rx clock cannot be recovered. |
|
RX power high |
Rx power is high. |
|
RX power low |
Rx power is low. |
|
TX not ready |
Tx is not ready. |
|
TX fault |
Tx fault. |
|
TX CDR loss of lock |
Tx clock cannot be recovered. |
|
TX bias high |
Tx bias current is high. |
|
TX bias low |
Tx bias current is low. |
|
TX power high |
Tx power is high. |
|
TX power low |
Tx power is low. |
|
Module not ready |
Module is not ready. |
|
APD supply fault |
APD supply fault. |
|
TEC fault |
TEC fault. |
|
Wavelength unlocked |
Wavelength of optical signal exceeds the manufacturer's tolerance. |
|
Temp high |
Temperature is high. |
|
Temp low |
Temperature is low. |
|
Voltage high |
Voltage is high. |
|
Voltage low |
Voltage is low. |
|
Transceiver info I/O error |
Transceiver information read and write error. |
|
Transceiver info checksum error |
Transceiver information checksum error. |
|
Transceiver type and port configuration mismatch |
The transceiver type does not match the port configuration. |
|
Transceiver type not supported by port hardware |
The transceiver type is not supported on the port. |
3. Swap the suspected transceiver module and a transceiver module that can correctly operate, and swap the interfaces.
4. If you are sure that the transceiver module fails, execute the display transceiver diagnosis command to collect the current values of the digital diagnosis parameters on the transceiver module and send them to H3C Support. The display transceiver diagnosis command applies to H3C transceiver modules and might not be able to display information about non-H3C transceiver modules.
<sysname>display transceiver diagnosis interface GigabitEthernet 1/0/17
GigabitEthernet1/0/17 transceiver diagnostic information:
Current diagnostic parameters:
Temp.(°C) Voltage(V) Bias(mA) RX power(dBm) TX power(dBm)
54 3.35 5.39 -5.91 -5.29
Alarm thresholds:
Temp.(°C) Voltage(V) Bias(mA) RX power(dBm) TX power(dBm)
High 73 3.80 11.00 0.00 0.00
Low -3 2.81 1.00 -16.99 -12.52
<sysname>
5. Display the electronic label information for the transceiver module. The Vendor Name field displays H3C for an H3C transceiver module. As a best practice, use only H3C transceiver modules.
<sysname>display transceiver manuinfo interface
GigabitEthernet1/0/16 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet1/0/17 transceiver manufacture information:
The transceiver does not support this function.
GigabitEthernet1/0/18 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet1/0/19 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet1/0/20 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet1/0/21 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet1/0/22 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet1/0/23 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/16 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/17 transceiver manufacture information:
The transceiver does not support this function.
GigabitEthernet2/0/18 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/19 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/20 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/21 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/22 transceiver manufacture information:
The transceiver is absent.
GigabitEthernet2/0/23 transceiver manufacture information:
The transceiver is absent.
6. If the issue persists, contact H3C Support.
Related commands
This section lists the commands that you might use for troubleshooting interfaces.
|
Command |
Description |
|
display current-configuration |
Displays the running configuration. You can display the running configuration for a specific interface. |
|
display interface |
Displays interface information, including the interface status and the incoming and outgoing traffic statistics. |
|
Displays transceiver alarms. |
|
|
display transceiver diagnosis |
Displays the current values of the digital diagnosis parameters on transceiver modules, including the temperature, voltage, bias current, incoming power, and outgoing power. |
|
display transceiver interface |
Displays the key parameters of transceiver modules. |
|
display transceiver manuinfo |
Displays electronic label information for transceiver modules to identify the vendors of the transceiver modules. |
Troubleshooting packet forwarding failures
Device ping failure from a directly connected PC
Symptom
The PC is connected to a service interface of the device through a network cable and is in the same subnet as the device. However, you cannot successfully ping the device from the PC.
Figure 3 Network diagram
Solution
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Network > Security Zones page.
3. Click the Edit icon for the target security zone.
4. Add the interface that connects the device to the PC as a member interface.
5. Click OK.
6. Access the Policies > Security Policies page.
7. On the Security Policies tab, click Create, and then click Create a policy.
8. Configure policy parameters as needed:
¡ Source zone—Select the zone to which the interface belongs as the source zone. In this example, the source zone is Trust.
¡ Name—Specify the policy name. In this example, the name is trust-local.
¡ Destination zone—Select Local as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of the PC as the source IP. In this example, the address is 10.1.1.2.
¡ Destination IPv4 address—Specify the IP address of the device as the destination IP. In this example, the address is 10.1.1.1.
For the device to access the PC, create a security policy to permit packets from the device to the PC.
¡ Name—Specify the policy name. In this example, the name is local-trust.
¡ Source zone—Select Local as the source zone.
¡ Destination zone—Select the zone to which the interface belongs as the destination zone. In this example, the destination zone is Trust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of the device as the source IP. In this example, the address is 10.1.1.1.
¡ Destination IPv4 address—Specify the IP address of the PC as the destination IP. In this example, the address is 10.1.1.2.
9. Click OK.
Connectivity failure between two PCs connected through the device
Symptom
Two PCs are connected through the device, and IP and route settings are configured correctly. However, the two PCs cannot reach each other.
Figure 4 Network diagram
Solution
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Network > Security Zones page.
3. Click the Edit icon for the target security zone.
4. Add the interface that connects the device to the PC as a member interface.
5. Click OK.
6. Repeat the previous steps to add the device's interface for the other PC to another security zone.
7. Access the Policies > Security Policies page.
8. On the Security Policies tab, click Create, and then click Create a policy. Create a security policy to permit packets from PC A to PC B.
9. Configure policy parameters as needed. As a best practice, specify exact match criteria.
¡ Name—Specify the policy name. In this example, the name is trust-untrust.
¡ Source zone—Select the zone to which the interface connecting PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the zone to which the interface connecting PC B belongs as the destination zone. In this example, the destination zone is Untrust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 10.1.1.2.
¡ Destination IPv4 address—Specify the IP address of PC B as the destination IP. In this example, the address is 20.1.1.1.
10. For the device to access the PC, create a security policy to permit packets from the device to the PC.
¡ Name—Specify the policy name. In this example, the name is untrust-trust.
¡ Source zone—Select the zone to which the interface connecting PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select the zone to which the interface connecting PC A belongs as the destination zone. In this example, the destination zone is Trust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 20.1.1.2.
¡ Destination IPv4 address—Specify the IP address of PC A as the destination IP. In this example, the address is 10.1.1.1.
11. Click OK.
Connectivity failure between PCs connected through the device in the same security zone
Symptom
Two PCs are connected through the device, and IP and route settings are configured correctly. The PCs are in the same security zone but cannot reach each other.
Figure 5 Network diagram
Solution
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies page.
3. On the Security Policies tab, click Create, and then click Create a policy.
4. Configure policy parameters as needed.
¡ Name—Specify the policy name. In this example, the name is trust-trust.
¡ Source zone—Select the zone to which the PCs belong as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the same zone as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP addresses of PC A and PC B as the source IPs. In this example, the addresses are 10.1.1.2 and 20.1.1.2 for PC A and PC B, respectively.
¡ Destination IPv4 address—Specify the IP addresses of PC B and PC A as the destination IPs. In this example, the addresses are 20.1.1.2 and 10.1.1.2 for PC B and PC A, respectively.
5. Click OK.
Ping or tracert operation failure
Symptom
The device fails to ping or trace route to a destination.
For example, all ICMP echo requests sent by device 192.168.20.14 to ping device 10.0.0.5 timed out and no replies were received.
<sysname> ping 10.0.0.5
PING 10.0.0.5 (10.0.0.5): 56 data bytes, press CTRL_C to break
Request time out
Request time out
Request time out
Request time out
Request time out
--- 10.0.0.5 ping statistics ---
5 packet(s) transmitted, 0 packet(s) received, 100.0% packet loss
Solution
To resolve the issue:
1. Execute the display security-zone command to verify that the input and output interfaces involved in packet forwarding have been added to a security zone.
<sysname>display security-zone
Name: Local
Members: None
Name: Trust
Members: GigabitEthernet1/0/8 Reth1
Name: DMZ
Members: None
Name: Untrust
Members: GigabitEthernet1/0/10 Reth2
Name: Management
Members: GigabitEthernet1/0/
2. Execute the display security-policy command to verify that security policies have been configured.
<sysname>display security-policy ip
Security-policy ip
rule 0 name 1
action pass
<sysname>display security-policy ipv6
Security-policy ipv6
rule 0 name IPv6
action pass
3. Identify the packet forwarding path and locate where the ICMP packets are lost on the path.
You can compare the ICMP packet statistics collected from the input and output interfaces of a node to identify packet loss. To clear history statistics for an interface, execute the reset counters interface command.
a. If no ICMP packets are received on the input interface, examine the adjacent upstream device for faults.
b. If the number of input ICMP packets matches the number of output ICMP packets, examine the adjacent downstream device for faults.
c. If no ICMP packets are forwarded on the output interface, proceed to the next step.
4. Execute the debugging aspf packet acl and debugging aspf event commands to identify if ICMP packet loss happens during the ASPF process. If Layer 2 ICMP packet forwarding is correct, execute the display ip statistics command to determine the cause for packet loss at Layer 3.
<sysname> display ip statistics
Input: sum 263207520 local 1772
bad protocol 0 bad format 0
bad checksum 0 bad options 0
Output: forwarding 24511617 local 476
dropped 21949 no route 156
compress fails 0
Fragment:input 0 output 0
dropped 0
fragmented 0 couldn't fragment 0
Reassembling:sum 0 timeouts 0
5. If the issue persists, contact Technical Support.
Ping operation failure across NAT
Symptom
A PC fails to ping another PC in a different subnet despite a successful NAT.
For example, PC1 10.1.1.1 pings PC2 220.1.1.2 across a device that translates PC1's IP address into 220.1.1.1. Although PC2 has received PC1's ICMP echo request, PC1 cannot receive an ICMP echo reply from PC2.
Solution
To resolve the issue:
1. Verify that the input and output interfaces of PC1 and PC2 have been added to security zones, and execute the display security-policy command to verify security policies have been configured.
<sysname> display security-policy ip
Security-policy ip
rule 0 name tom-tom1
action pass
counting enable
source-zone tom
destination-zone tom1
2. Execute the display ip routing-table command on the device to verify that the device RIB contains a route to PC1.
<sysname> display ip routing-table 10.1.1.0
If no routes to PC1 exist, examine the routing protocol configurations and verify that the protocols are operating correctly.
3. Execute the display fib command on the device to verify that the device FIB contains a route to PC1.
<sysname> display fib 10.1.1.0
If the RIB contains a route to PC1 but the FIB does not, contact Technical Support.
4. Execute the display arp command on the device to verify that the device ARP table contains an entry for the IP address of PC1 (10.1.1.1).
<sysname> display arp 10.1.1.1
5. Execute the display session command on the device to verify that the session is established correctly.
6. Enable security policy packet debugging on the device to view packet denial statistics.
If an ASPF policy is applied, you must configure detect icmp for the policy or configure security policies to permit return packets from the destination zone to the source zone. If you do not do so, the device denies return packets.
<sysname> debugging security-policy packet ip acl ?
INTEGER<2000-2999> Specify a basic ACL
INTEGER<3000-3999> Specify an advanced ACL
Example output for packet denial is as follows:
*Jul 21 11:00:00:838 2017 F1090-IRF FILTER/7/PACKET: -Context=1; The packet is deny. Src-Zone=tom1, Dst-Zone=tom;If-In=, If-Out=Reth11(134); Packet Info:Src-IP=220.1.1.2, Dst-IP=10.1.1.1, VPN-Instance=,Src-Port=1024, Dst-Port=1025, Protocol= UDP(17), ACL=none, Rule-ID=0.
7. Verify that the OpenFlow flow table has been deployed correctly.
a. Verify that the OpenFlow flow table has been deployed to the interface module.
# Enable outbound static NAT.
[SYSNAME] nat static outbound 10.1.1.1 220.1.1.1
# Enable static NAT on the interface.
# Check whether the OpenFlow flow table has been deployed to the interface module correctly.
[SYSNAME-probe] display system internal openflow instance inner-redirect flow-table
Instance 4097 Flow Table Information:
Table 200 information:
Table type: Extensibility, flow entry count: 25, total flow entry count: 25
Flow entry rule 6 information:
cookie: 0x0, priority: 7861, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG11
Ethernet type: 0x0800
IP Range: IPv4 destination address from 220.1.1.1 to 220.1.1.1
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Flow entry rule 7 information:
cookie: 0x0, priority: 7840, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 source address from 10.10.1.1 to 10.10.1.1
VRF index: 0
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Flow entry rule 8 information:
cookie: 0x0, priority: 7841, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 destination address from 10.10.1.1 to 10.10.1.1
VRF index: 0
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Anly missing of the preceding flow entry rules will result in forwarding errors.
b. Check whether the OpenFlow flow table has been deployed to the service module correctly.
[SYSNAME-probe]display system internal openflow instance inner flow-table
Instance 4096 Flow Table Information:
Table 200 information:
Table type: Extensibility, flow entry count: 27, total flow entry count: 27
Flow entry rule 6 information:
cookie: 0x0, priority: 7860, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 destination address from 220.1.1.1 to 220.1.1.1
VRF index: 0
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Flow entry rule 7 information:
cookie: 0x0, priority: 7840, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 source address from 10.10.1.1 to 10.10.1.1
VRF index: 0
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Flow entry rule 8 information:
cookie: 0x0, priority: 7841, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 destination address from 10.10.1.1 to 10.10.1.1
VRF index: 0
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
8. If the issue persists, contact Technical Support.
Packet loss
Symptom
Packet loss occurs during packet forwarding.
Solution
To resolve the issue:
1. Execute the debugging security-policy packet command and determine if the packet loss is caused by incorrect security policy configurations.
<sysname>*Jan 13 16:06:32:298 2020 8350-2 FILTER/7/PACKET: -Context=1; The packet is denied. Src-Zone=Untrust, Dst-Zone=Trust;If-In=GigabitEthernet1/0/14(17), If-Out=GigabitEthernet1/0/10(13); Packet Info:Src-IP=10.1.1.3, Dst-IP=100.1.1.3, VPN-Instance=, Src-MacAddr=3897-d6a9-1e58,Src-Port=1024, Dst-Port=1024, Protocol=TCP(6), Application=general_tcp(2086),Terminal=invalid(0), SecurityPolicy=r0, Rule-ID=0.
If the output from the command contains The packet is denied, the packet loss is caused by incorrect security policy configurations.
2. Execute the debugging ip packet command to view information about lost packets.
Table 5 Command output description
|
Field |
Description |
|
Send the packet. |
|
|
Receive the packet. |
|
|
Deliver IP packets to the upper layer. |
|
|
Interface that received or sent the packet. |
|
|
IP version of the packet. |
|
|
Header length of the packet. |
|
|
Service type of the packet. |
|
|
Total length of the packet. |
|
|
ID of the packet. |
|
|
Fragmentation offset of the packet. |
|
|
Time to live of the packet. |
|
|
Protocol field of the packet. |
|
|
Header checksum of the packet. |
|
|
Source address of the packet. |
|
|
Destination address of the packet. |
|
|
Sending the packet from local at interface-type interface-number |
Send the packet from the local interface. |
|
Receiving IP packet from interface-type interface-number |
Receive the packet from the interface. |
|
Deliver the received packet to the upper layer. |
3. Execute the debugging ip error and debug ip info acl commands to determine the cause for packet loss.
Table 6 Possible causes for packet loss
|
Field |
Description |
|
The number of reassembly queues has exceeded the limit. |
|
|
The number of segments in the reassembly queue has exceeded the limit. |
|
|
Reassembly fails. |
|
|
Fail to obtain the forwarding control block. |
|
|
Release MBUF! Phase Num is num, Service ID is id, Bitmap is %#lx! |
MBUF is released. The num, id, and %#lx arguments represent the phase number, service ID, and bitmap, respectively. |
|
Forwarding of broadcast packets to the output interface subnet is not allowed. |
|
|
Error output interface is assigned. |
4. If the issue persists, contact Technical Support.
Related commands
|
Command |
Description |
|
display arp |
Displays ARP entries. |
|
display current-configuration | include lsr-id |
Displays the current MPLS LSR ID. |
|
display current-configuration configuration mpls-ldp |
Displays the currently effective MPLS LDP configuration. Use thid command to identify whether the LDPs are consistent in the MD5 password. |
|
display fib |
Displays FIB entries. |
|
display interface |
Displays interface information. |
|
display ip interface brief |
Displays brief IP configuration for Layer 3 interfaces. |
|
display ip routing-table |
Displays routing table information. |
|
display session |
Displays session information. |
|
display this |
Displays the running configuration in the current view. |
|
interface |
Enters interface view. |
|
display system internal openflow instance |
Displays information about OpenFlow flow entries. |
|
display nat outbound |
Displays information about outbound dynamic NAT. |
Troubleshooting IRF
IRF fabric establishment failure
Symptom
An IRF fabric cannot be established.
Solution
This issue is typically caused by configuration errors.
To resolve the issue:
1. Check whether the member devices are consistent in the device model, MPU model, and software version.
Execute the display version and display system internal version commands to verify that the member devices are the same model and run the same software version.
¡ If the models are different, use the same model of devices.
¡ If the software versions are different, upgrade the software to the same version.
<sysname>display version
H3C Comware Software, Version 7.1.064, Release 9071P1313
Copyright (c) 2004-2022 New H3C Technologies Co., Ltd. All rights reserved.
H3C SecPath M9000-AI-E8 uptime is 0 weeks, 4 days, 22 hours, 6 minutes
Last reboot reason : Cold reboot
Boot image: flash:/M9000E-CMW710-BOOT-R9071P1313.bin
Boot image version: 7.1.064, Release 9071P1313
Compiled Sep 07 2022 15:00:00
System image: flash:/M9000E-CMW710-SYSTEM-R9071P1313.bin
System image version: 7.1.064, Release 9071P1313
Compiled Sep 07 2022 15:00:00
LPU 2:
Uptime is 0 weeks,4 days,22 hours,3 minutes
H3C SecPath M9000-AI-E8 LPU with 1 ARM Processor
BOARD TYPE: NSQM5MBSHA1
DRAM: 2048M bytes
PCB 1 Version: VER.A
SUBCARD 1 PCB Version:VER.A
SUBCARD 2 PCB Version:VER.A
Bootrom Version: 100
CPLD 1 Version: 001
SUBCARD 1 CPLD Version:002
SUBCARD 2 CPLD Version:001
Release Version: H3C SecPath M9000-AI-E8-9071P1313
Patch Version : None
Reboot Cause : ColdReboot
PowChip Version: 001
SLOT 2 CPU 1
CPU type: Multi-core CPU
DDR4 : 32752M bytes
FLASH: 7296M bytes
Board PCB Version: Ver.A
CPLD Version: 2.0
Release Version: SecBlade AFC Enhanced-9071P1313
Basic BootWare Version:1.04
Extend BootWare Version:1.04
Reboot Cause: Warm reboot
SLOT 2 CPU 2
CPU type: Multi-core CPU
DDR4 : 32752M bytes
FLASH: 7296M bytes
Board PCB Version: Ver.A
CPLD Version: 2.0
Release Version: SecBlade FW Enhanced-9071P1313
Basic BootWare Version:1.04
Extend BootWare Version:1.04
Reboot Cause: Warm reboot
MPU(M) 4:
Uptime is 0 weeks,4 days,22 hours,6 minutes
H3C SecPath M9000-AI-E8 MPU(M) with 1 XLP316 Processor
BOARD TYPE: NSQM5SUP08A1
DRAM: 8192M bytes
FLASH: 1024M bytes
PCB 1 Version: VER.A
Bootrom Version: 158
CPLD 1 Version: 003
CPLD 2 Version: 001
Release Version: H3C SecPath M9000-AI-E8-9071P1313
Patch Version : None
Reboot Cause : ColdReboot
NPU 6:
BOARD TYPE: NSQM5FAB08A1
PCB Version: VER.A
CPLD Version: 200
NPU 7:
BOARD TYPE: NSQM5FAB08A1
PCB Version: VER.A
CPLD Version: 200
2. Verify that the number of member devices does not exceed the maximum allowed number.
An IRF fabric can contain a maximum of two member devices currently.
3. Verify that the member ID of each member device is unique:
a. Execute the display irf command to view the member ID of each member device.
<sysname> display irf
MemberID Role Priority CPU-Mac Description
*+1 Master 1 00ff-fbec-b003 ---
--------------------------------------------------
* indicates the device is the master.
+ indicates the device through which the user logs in.
The bridge MAC of the IRF is: 00ff-fbec-b001
Auto upgrade : yes
Mac persistent : 6 min
Domain ID : 0
b. If the member IDs are not unique, execute the irf member renumber command to change the member ID of one member device.
4. Verify that the physical interfaces bound to IRF ports can act as IRF physical interfaces. For more information, see IRF configuration in the configuration guides.
5. Verify that the IRF port bindings and physical IRF link connections are correct:
|
|
IMPORTANT: When you connect two neighboring IRF members, you must connect the physical interfaces of IRF-port 1 on one member to the physical interfaces of IRF-port 2 on the other. |
a. Execute the display irf configuration command on each member device, and check the IRF-Port1 and IRF-Port2 fields for IRF port bindings.
b. Verify that the physical IRF connections are consistent with the IRF port bindings.
c. If there are binding errors or connection inconsistencies, reconfigure the IRF port bindings or reconnect the IRF physical interfaces.
6. Verify that a minimum of one IRF physical link is up:
a. Execute the display interface command to verify that the IRF physical interfaces are up.
<sysname> display interface gigabitethernet 1/0/10
GigabitEthernet1/0/10
Current state: UP
Line protocol state: UP
Description: GigabitEthernet1/0/10 Interface
Bandwidth: 1000000kbps
Maximum Transmit Unit: 1500
Internet protocol processing: disabled
IP Packet Frame Type:PKTFMT_ETHNT_2, Hardware Address: 8042-0000-560a
IPv6 Packet Frame Type:PKTFMT_ETHNT_2, Hardware Address: 8042-0000-560a
Media type is twisted pair
Port hardware type is 1000_BASE_T
Last clearing of counters: Never
Peak value of input: 0 bytes/sec, at 2013-12-13 15:15:02
Peak value of output: 0 bytes/sec, at 2013-12-13 15:15:02
Last 300 seconds input: 0 packets/sec 0 bytes/sec
Last 300 seconds output: 0 packets/sec 0 bytes/sec
b. If no IRF physical links are up, locate the issue and bring up a minimum of one IRF physical link. An IRF physical link comes up when the IRF physical interfaces at both ends of the link come up. To bring up an interface, see "Interface fails to come up."
7. Verify that the IRF ports of the member devices are consistent in binding mode.
[SYSNAME] irf-port 1/2
[SYSNAME-irf-port1/2] display this
irf-port 1/2
port group interface Ten-GigabitEthernet1/3/0/1 mode enhanced
8. If the issue persists, contact H3C Support.
IRF split
Symptom
An IRF fabric splits.
Solution
To resolve the issue:
1. Search the log messages for the IRF port down event. This event helps you determine the time when the IRF fabric split.
%Jun 26 10:13:46:233 2013 H3C STM/2/STM_LINK_STATUS_TIMEOUT: IRF port 1 is down because heartbeat timed out.
%Jun 26 10:13:46:436 2013 H3C STM/3/STM_LINK_STATUS_DOWN: -MDC=1; IRF port 2 is down.
2. Verify that the interface module where each IRF physical port resides is in normal status. If it is abnormal, see "Abnormal card state or card failure" in "Troubleshooting hardware" to troubleshoot.
<sysname> display device
Chassis Slot Type State Subslot Soft Ver Patch Ver
2 0 NSQ1GT48EA0 Normal 0 M9014-9153P22 None
2 1 NONE Absent 0 NONE None
2 2 NONE Absent 0 NONE None
2 3 NSQ1TGS8EA0 Normal 0 M9014-9153P22 None
2 4 NSQ1FWCEA0 Normal 0 M9014-9153P22 None
2 5 NONE Absent 0 NONE None
2 6 NSQ1SUPB0 Master 0 M9014-9153P22 None
2 7 NSQ1SUPB0 Standby 0 M9014-9153P22 None
2 8 NONE Absent 0 NONE None
2 9 NONE Absent 0 NONE None
2 10 NSQ1FWCEA0 Normal 0 M9014-9153P22 None
2 11 NONE Absent 0 NONE None
2 12 NONE Absent 0 NONE None
2 13 LSU1GP24TXEB0 Normal 0 M9014-9153P22 None
2 14 NONE Absent 0 NONE None
2 15 NSQ1FAB12D0 Normal 0 M9014-9153P22 None
2 16 NSQ1FAB12D0 Normal 0 M9014-9153P22 None
2 17 NSQ1FAB12D0 Normal 0 M9014-9153P22 None
3. Verify that the IRF physical interfaces are operating correctly.
a. Execute the display interface command to identify the state of the IRF physical interfaces. If an IRF physical interface is not up or has other issues, locate and resolve the issue as described in "Troubleshooting interfaces."
<sysname> display interface gigabitethernet 1/0/10
GigabitEthernet1/0/10 current state: UP
Line protocol current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 0000-e80d-c000
Description: GigabitEthernet2/6/0/1 Interface
Loopback is not set
Media type is optical fiber, Port hardware type is 1000_BASE_SX_SFP
...
b. If the issue persists, remove faulty IRF physical interfaces from the IRF ports and bind new IRF physical interfaces to the IRF ports.
4. Remove hardware issues that might cause recurring IRF split events:
a. Execute the display version command to identify the uptime of IRF member devices and cards (if any) that have IRF links.
b. Compare the uptime of IRF member devices and cards (if any) to determine whether a member device or card had rebooted before the IRF split.
c. If the IRF split is caused by a device or card (if any) reboot or power failure, locate and resolve the issue as described in "Troubleshooting hardware."
<sysname>dis version
H3C Comware Software, Version 7.1.064, Release 9071P1313
Copyright (c) 2004-2022 New H3C Technologies Co., Ltd. All rights reserved.
H3C SecPath M9000-AI-E8 uptime is 0 weeks, 4 days, 22 hours, 6 minutes
Last reboot reason : Cold reboot
Boot image: flash:/M9000E-CMW710-BOOT-R9071P1313.bin
Boot image version: 7.1.064, Release 9071P1313
Compiled Sep 07 2022 15:00:00
System image: flash:/M9000E-CMW710-SYSTEM-R9071P1313.bin
System image version: 7.1.064, Release 9071P1313
Compiled Sep 07 2022 15:00:00
LPU 2:
Uptime is 0 weeks,4 days,22 hours,3 minutes
H3C SecPath M9000-AI-E8 LPU with 1 ARM Processor
BOARD TYPE: NSQM5MBSHA1
DRAM: 2048M bytes
PCB 1 Version: VER.A
SUBCARD 1 PCB Version:VER.A
SUBCARD 2 PCB Version:VER.A
Bootrom Version: 100
CPLD 1 Version: 001
SUBCARD 1 CPLD Version:002
SUBCARD 2 CPLD Version:001
Release Version: H3C SecPath M9000-AI-E8-9071P1313
Patch Version : None
Reboot Cause : ColdReboot
PowChip Version: 001
SLOT 2 CPU 1
CPU type: Multi-core CPU
DDR4 : 32752M bytes
FLASH: 7296M bytes
Board PCB Version: Ver.A
CPLD Version: 2.0
Release Version: SecBlade AFC Enhanced-9071P1313
Basic BootWare Version:1.04
Extend BootWare Version:1.04
Reboot Cause: Warm reboot
SLOT 2 CPU 2
CPU type: Multi-core CPU
DDR4 : 32752M bytes
FLASH: 7296M bytes
Board PCB Version: Ver.A
CPLD Version: 2.0
Release Version: SecBlade FW Enhanced-9071P1313
Basic BootWare Version:1.04
Extend BootWare Version:1.04
Reboot Cause: Warm reboot
MPU(M) 4:
Uptime is 0 weeks,4 days,22 hours,6 minutes
H3C SecPath M9000-AI-E8 MPU(M) with 1 XLP316 Processor
BOARD TYPE: NSQM5SUP08A1
DRAM: 8192M bytes
FLASH: 1024M bytes
PCB 1 Version: VER.A
Bootrom Version: 158
CPLD 1 Version: 003
CPLD 2 Version: 001
Release Version: H3C SecPath M9000-AI-E8-9071P1313
Patch Version : None
Reboot Cause : ColdReboot
NPU 6:
BOARD TYPE: NSQM5FAB08A1
PCB Version: VER.A
CPLD Version: 200
NPU 7:
BOARD TYPE: NSQM5FAB08A1
PCB Version: VER.A
CPLD Version: 200
5. If the IRF split issue persists, collect device diagnostic information, and then send the information to H3C Support.
Related commands
|
Command |
Description |
|
display interface |
Displays interface information. Use this command to identify whether an IRF physical interface is up. |
|
display irf |
Displays IRF fabric information, including the member ID, role, priority, bridge MAC address, and description of each IRF member. |
|
display irf configuration |
Displays basic IRF settings, including the current member ID, new member ID, and physical interfaces bound to the IRF ports on each IRF member device. The new member IDs take effect at reboot. Use this command to identify whether the physical interfaces of IRF-port 1 on one member are connected to the physical interfaces of IRF-port 2 on the other member. |
|
display current-configuration |
Displays the currently effective configuration. Use this command to identify whether the member devices are consistent in the IRF physical interface binding mode. |
|
display version display system internal version |
Displays system version information. Use this command to identify whether the member devices are the same model and run the same software version. |
Troubleshooting RBM
RBM system setup failure
Symptom
Two devices cannot form an RBM system.
Solution
To resolve the issue:
1. Execute the display version command to verify that the member devices of the RBM system are the same model.
<sysname>display version
H3C Comware Software, Version 7.1.064, Ess 9671P18
Copyright (c) 2004-2022 New H3C Technologies Co., Ltd. All rights reserved.
H3C SecPath M9000-X06 uptime is 0 weeks, 1 day, 22 hours, 39 minutes
Last reboot reason : Cold reboot
Boot image: flash:/M9000X-CMW710-BOOT-E9671P18.bin
Boot image version: 7.1.064, Ess 9671P18
Compiled Dec 14 2022 15:00:00
System image: flash:/M9000X-CMW710-SYSTEM-E9671P18.bin
System image version: 7.1.064, Ess 9671P18
Compiled Dec 14 2022 15:00:00
LPU 1:
Uptime is 0 weeks,1 day,22 hours,35 minutes
H3C SecPath M9000-X06 LPU with 1 ARM Processor
BOARD TYPE: NSQM7MBSHA0
DRAM: 2048M bytes
PCB 1 Version: VER.A
SUBCARD 1 PCB Version:VER.A
SUBCARD 2 PCB Version:VER.A
Bootrom Version: 101
CPLD 1 Version: 001
SUBCARD 1 CPLD Version:002
SUBCARD 2 CPLD Version:002
Release Version: H3C SecPath M9000-X06-9671P18
Patch Version : None
Reboot Cause : ColdReboot
PowChip Version: 001
SLOT 1 CPU 3
CPU type: Multi-core CPU
DDR4 : 98304M bytes
FLASH: 7281M bytes
Board PCB1 Version: Ver.A
Board PCB2 Version: Ver.A
BMC Version: 2.24.03
CPLD1 Version: 2.0
CPLD2 Version: 1.0
CPLD3 Version: 3.0
Release Version: SecBlade FW Enhanced-9671P18
FPGA Version: B6001
FPGA DATE: 2022.12.08
Basic BootWare Version:1.06
Extend BootWare Version:1.06
Reboot Cause: User reboot
MPU(M) 4:
Uptime is 0 weeks,1 day,22 hours,39 minutes
H3C SecPath M9000-X06 MPU(M) with 1 XLP316 Processor
BOARD TYPE: NSQM7SUPB0
DRAM: 8192M bytes
FLASH: 1024M bytes
PCB 1 Version: VER.A
Bootrom Version: 100
CPLD 1 Version: 001
CPLD 2 Version: 004
Release Version: H3C SecPath M9000-X06-9671P18
Patch Version : None
Reboot Cause : ColdReboot
NPU 9:
BOARD TYPE: NSQM7FAB06A0
PCB Version: VER.A
CPLD Version: 200
2. Verify that the RBM system has only two member devices.
3. Execute the display irf command to verify that the member devices use unique IRF member IDs. If the member devices use the same IRF member ID, use the irf member command to modify the IRF member ID for one of the member devices.
<sysname>display irf
MemberID Role Priority CPU-Mac Description
*+1 Master 1 80e4-55d8-54ae ---
--------------------------------------------------
* indicates the device is the master.
+ indicates the device through which the user logs in.
The bridge MAC of the IRF is: 80e4-55d8-54ac
Auto upgrade : yes
Mac persistent : 6 min
Domain ID : 0
4. Execute the display interface brief command to verify that the RBM data and control channel settings are consistent on the member devices.
5. Verify that the member devices use the same software version.
<sysname>display version
H3C Comware Software, Version 7.1.064, Ess 9671P18
Copyright (c) 2004-2022 New H3C Technologies Co., Ltd. All rights reserved.
H3C SecPath M9000-X06 uptime is 0 weeks, 1 day, 22 hours, 39 minutes
Last reboot reason : Cold reboot
Boot image: flash:/M9000X-CMW710-BOOT-E9671P18.bin
Boot image version: 7.1.064, Ess 9671P18
Compiled Dec 14 2022 15:00:00
System image: flash:/M9000X-CMW710-SYSTEM-E9671P18.bin
System image version: 7.1.064, Ess 9671P18
Compiled Dec 14 2022 15:00:00
LPU 1:
Uptime is 0 weeks,1 day,22 hours,35 minutes
H3C SecPath M9000-X06 LPU with 1 ARM Processor
BOARD TYPE: NSQM7MBSHA0
DRAM: 2048M bytes
PCB 1 Version: VER.A
SUBCARD 1 PCB Version:VER.A
SUBCARD 2 PCB Version:VER.A
Bootrom Version: 101
CPLD 1 Version: 001
SUBCARD 1 CPLD Version:002
SUBCARD 2 CPLD Version:002
Release Version: H3C SecPath M9000-X06-9671P18
Patch Version : None
Reboot Cause : ColdReboot
PowChip Version: 001
SLOT 1 CPU 3
CPU type: Multi-core CPU
DDR4 : 98304M bytes
FLASH: 7281M bytes
Board PCB1 Version: Ver.A
Board PCB2 Version: Ver.A
BMC Version: 2.24.03
CPLD1 Version: 2.0
CPLD2 Version: 1.0
CPLD3 Version: 3.0
Release Version: SecBlade FW Enhanced-9671P18
FPGA Version: B6001
FPGA DATE: 2022.12.08
Basic BootWare Version:1.06
Extend BootWare Version:1.06
Reboot Cause: User reboot
MPU(M) 4:
Uptime is 0 weeks,1 day,22 hours,39 minutes
H3C SecPath M9000-X06 MPU(M) with 1 XLP316 Processor
BOARD TYPE: NSQM7SUPB0
DRAM: 8192M bytes
FLASH: 1024M bytes
PCB 1 Version: VER.A
Bootrom Version: 100
CPLD 1 Version: 001
CPLD 2 Version: 004
Release Version: H3C SecPath M9000-X06-9671P18
Patch Version : None
Reboot Cause : ColdReboot
NPU 9:
BOARD TYPE: NSQM7FAB06A0
PCB Version: VER.A
CPLD Version: 200
6. Verify that the interfaces used to set up the RBM control and data channels are up. If an interface is down, troubleshoot the interface as described in "Interface fails to come up."
<sysname>display interface GigabitEthernet 1/0/1
GigabitEthernet1/0/1
Current state: UP
Line protocol state: UP
Description: GigabitEthernet1/0/1 Interface
Bandwidth: 1000000 kbps
Maximum transmission unit: 1500
Allow jumbo frames to pass
Broadcast max-ratio: 100%
Multicast max-ratio: 100%
Unicast max-ratio: 100%
Internet protocol processing: Disabled
IP packet frame type: Ethernet II, hardware address: 80e4-55d8-54b3
IPv6 packet frame type: Ethernet II, hardware address: 80e4-55d8-54b3
Media type is twisted pair, loopback not set, promiscuous mode not set
1000Mb/s, Full-duplex, link type is autonegotiation
Output flow-control is disabled, input flow-control is disabled
Last link flapping: 1 days 17 hours 29 minutes
Last clearing of counters: Never
Current system time:2021-02-01 08:42:30 Beijing+08:00:00
Last time when physical state changed to up:2021-01-30 15:12:46 Beijing+08:00:00
Last time when physical state changed to down:2021-01-30 15:12:08 Beijing+08:00:00
Peak input rate: 8499998 bytes/sec, at 2021-01-30 15:18:39
Peak output rate: 5172061 bytes/sec, at 2021-01-30 15:12:53
Last 300 second input: 0 packets/sec 22 bytes/sec 0%
Last 300 second output: 0 packets/sec 25 bytes/sec 0%
7. Verify that the member devices use the same destination port to set up the RBM control channel and the RBM control channel is up.
RBM_P[F1090]display remote-backup-group status
Remote backup group information:
Backup mode: Dual-active
Device role: Primary
Data channel interface: Route-Aggregation64
Local IPv6: 100::1
Remote IPv6: 100::2 Destination port: 60064
Control channel status: Connected
Hot backup status:Enabled
Auto configuration synchronization: Enable
Configuration consistency check interval: 1 hour
Delay-time: 1 min
RBM system split
Symptom
An RBM system splits unexpectedly.
Solution
To resolve the issue:
1. Check log messages for the RBM system split time, which is the time when the interfaces used by RBM went down.
RBM_P<Device-VRRP-ZHU-1>%Feb 1 07:57:49:310 2021 F1010-VRRP-ZHU-1 LLDP/6/LLDP_DELETE_NEIGHBOR: Nearest bridge agent neighbor deleted
on port GigabitEthernet1/0/7 (IfIndex 8), neighbor's chassis ID is d461-fe39-d20c, port ID is GigabitEthernet1/0/7.
%Feb 1 07:57:50:487 2021 F1010-VRRP-ZHU-1 IFNET/3/PHY_UPDOWN: Physical state on the interface GigabitEthernet1/0/7 changed to down.
%Feb 1 07:57:50:487 2021 F1010-VRRP-ZHU-1 IFNET/5/LINK_UPDOWN: Line protocol state on the interface GigabitEthernet1/0/7 changed to
down.
%Feb 1 07:58:00:269 2021 F1010-VRRP-ZHU-1 RBM/6/RBM_CHANNEL: Local IPv6=202::1, remote IPv6=202::2, status=Disconnected
2. Verify that the physical interfaces used by RBM are operating correctly. If an interface is abnormal, troubleshoot it as described in "Troubleshooting interfaces."
3. Check interface information for the cause of DR system split.
RBM_P<Device-VRRP-ZHU-1>display interface GigabitEthernet 1/0/7
GigabitEthernet1/0/7
Current state: UP
Line protocol state: UP
Description: link-f1010-bei
Bandwidth: 1000000 kbps
Maximum transmission unit: 1500
Allow jumbo frames to pass
Broadcast max-ratio: 100%
Multicast max-ratio: 100%
Unicast max-ratio: 100%
Internet address: 202.1.1.1/24 (Primary)
IP packet frame type: Ethernet II, hardware address: e8f7-24d9-2875
IPv6 packet frame type: Ethernet II, hardware address: e8f7-24d9-2875
Media type is twisted pair, loopback not set, promiscuous mode not set
1000Mb/s, Full-duplex, link type is autonegotiation
Output flow-control is disabled, input flow-control is disabled
Output queue - Urgent queuing: Size/Length/Discards 0/1024/0
Output queue - Protocol queuing: Size/Length/Discards 0/500/0
Output queue - FIFO queuing: Size/Length/Discards 0/75/0
Last link flapping: 0 hours 0 minutes 19 seconds
Last clearing of counters: Never
Current system time:2021-02-01 08:00:09
Last time when physical state changed to up:2021-02-01 07:59:51
Last time when physical state changed to down:2021-02-01 07:57:50
Peak input rate: 1694290 bytes/sec, at 2021-01-30 14:35:26
Peak output rate: 6245465 bytes/sec, at 2021-01-30 14:40:01
Last 300 second input: 1 packets/sec 132 bytes/sec 0%
Last 300 second output: 1 packets/sec 132 bytes/sec 0%
Input (total): 2404856 packets, 808021430 bytes
4. Check the device uptime and log messages for device reboot records or reboot records for the interface modules that provide interfaces for setting up the RBM control channel. If an interface module or a member device reboots when the DR system splits, check for power supply faults.
5. Replace failed transceiver modules or the interfaces used for setting up the RBM control channel to verify that the member devices can form an RBM system again.
6. If the issue persists, collect information about the member devices and contact H3C Support.
Troubleshooting RBM dynamic routing issues
RBM switchover not triggered upon uplink or downlink interface failure
Symptom
Traffic is still sent to an RBM member device for forwarding after its uplink or downlink interface fails.
Solution
To resolve the issue:
1. Log in to the RBM member devices and verify that they have the same number of service modules.
2. Configure the primary member device as follows:
RBM_P[M9016_1-remote-backup-group] track 1 interface Route-Aggregation1
RBM_P[M9016_1-remote-backup-group] track 2 interface Route-Aggregation11
RBM_P[M9016_1-remote-backup-group] display this
#
remote-backup group
backup-mode dual-active
data-channel interface Route-Aggregation1000
delay-time 1
adjust-cost bgp enable absolute 10000
adjust-cost ospf enable absolute 10000
adjust-cost ospfv3 enable absolute 10000
track 1
track 2
local-ip 192.168.195.9
remote-ip 192.168.195.10
device-role primary
3. Configure the secondary member device as follows:
RBM_S[M9016_2-remote-backup-group] track 1 interface Route-Aggregation1
RBM_S[M9016_2-remote-backup-group] track 2 interface Route-Aggregation11
RBM_S[M9016_2-remote-backup-group] display this
#
remote-backup group
backup-mode dual-active
data-channel interface Route-Aggregation1000
delay-time 1
adjust-cost bgp enable absolute 10000
adjust-cost ospf enable absolute 10000
adjust-cost ospfv3 enable absolute 10000
track 1
track 2
local-ip 192.168.195.10
remote-ip 192.168.195.9
device-role secondary
Inconsistent ACL configuration between the RBM member devices
Symptom
The RBM member devices have inconsistent ACL configuration, such as the following:
RBM_P[M9016_1]%Dec 17 14:25:43:191 2020 M9016_1 RBM/6/RBM_CFG_COMPARE_START: Started configuration consistency check.
%Dec 17 14:25:44:775 2020 M9016_1 RBM/6/RBM_CFG_COMPARE_RESULT: The following modules have inconsistent configuration: acl.
%Dec 17 14:25:44:775 2020 M9016_1 RBM/6/RBM_CFG_COMPARE_FINISH: Finished configuration consistency check.
Solution
To resolve the issue:
· If an ACL exists only on the secondary member device, perform the following tasks:
¡ To retain the ACL, create it on the primary member device, and save the running configuration of both RBM member devices.
¡ To delete the ACL, execute the configuration manual-sync command on the primary member device, and save the running configuration of both RBM member devices.
· If an ACL exists only on the primary member device, perform the following tasks:
¡ To retain the ACL, execute the configuration manual-sync command on the primary member device, and save the running configuration of both RBM member devices.
¡ To delete the ACL, delete it from the primary member device, execute the configuration manual-sync command on the primary member device, and save the running configuration of both RBM member devices.
Troubleshooting hot backup
A Reth interface is not pingable if not assigned to redundancy group
Symptom
A device cannot ping a directly connected Reth interface when the Reth interface is not in any redundancy group.
Solution
To resolve the issue:
1. Verify that the member interfaces of the Reth interface can correctly receive and send packets:
a. Execute the debugging ethernet packet command to debug packet transmission on the Reth interface and remove errors based on the command output.
debugging ethernet packet interface Reth 1
b. Execute the debugging arp error command to debug ARP learning and remove errors based on the command output.
debugging arp error
c. Execute the debugging ip error command to debug IP forwarding and remove errors based on the command output.
debugging ip error
d. Execute the display ethernet statistics command on both member devices of the hot backup system and identify whether the number of error packets increases with the number of packets transmitted.
[H3C] display ethernet statistics slot 1
ETH receive packet statistics:
Totalnum : 1000888 ETHIINum : 1000888
SNAPNum : 0 RAWNum : 0
LLCNum : 0 UnknownNum : 0
ForwardNum : 884856 ARP : 0
MPLS : 0 ISIS : 0
ISIS2 : 0 IP : 0
IPV6 : 0
ETH receive error statistics:
NullPoint : 0 ErrIfindex : 3
ErrIfcb : 0 IfShut : 5
ErrAnalyse : 0 ErrSrcMAC : 0
ErrHdrLen : 0
ETH send packet statistics:
L3OutNum : 325126 VLANOutNum : 0
FastOutNum : 92115615 L2OutNum : 0
ETH send error statistics:
MbufRelayNum : 0 NullMbuf : 0
ErrAdjFwd : 0 ErrPrepend : 0
ErrHdrLen : 0 ErrPad : 0
ErrQosTrs : 0 ErrVLANTrs : 0
ErrEncap : 287 ErrTagVLAN : 0
IfShut : 0 IfErr : 0
2. If the Reth interface cannot receive or send packets, verify that it is correctly configured:
a. Execute the display reth interface command and check the Physical status and Forwarding status fields.
<sysname>display reth interface Reth 1
Reth1 :
Redundancy group : fqs
Member Physical status Forwarding status Presence status
GE1/1/1.500 UP Active Normal
GE2/0/1.500 UP Inactive Normal
b. If both member interfaces of the Reth interface are inactive or down, locate the cause and resolve it.
c. If the member interfaces are operating correctly, check whether they have correct ARP entries. If the member interfaces are subinterfaces, the ARP entries must contain correct VLAN IDs.
d. Restart the Reth interface to verify that the ARP entries can be refreshed correctly.
e. Check the packet statistics about the physical member interfaces to verify that the driver correctly sends packets to the CPU.
3. If the issue persists, contact H3C Support.
Stateful failover failure
Symptom
Figure 6 Network diagram
Network configuration
Device 1 and Device 2 form an IRF system. Reth 1 is an uplink interface with members Route-Aggregation1 and Route-Aggregation2. Route-Aggregation1 has a higher priority.
Reth 2 is a downlink interface with members Route-Aggregation3 and Route-Aggregation4. Route-Aggregation3 has a higher priority.
Both Reth 1 and Reth 2 have IP addresses configured. Redundancy group 1 contains Reth 1 and Reth 2.
Procedure
interface Reth 1
ip address 100.1.1.1 255.255.255.0
member interface Route-Aggregation1 priority 100
member interface Route-Aggregation2 priority 1
interface Reth 2
ip address 100.1.1.1 255.255.255.0
member interface Route-Aggregation3 priority 100
member interface Route-Aggregation4 priority 1
track 11 interface Route-Aggregation1
track 12 interface Route-Aggregation2
track 13 interface Route-Aggregation3
track 14 interface Route-Aggregation4
redundancy group 1
member interface Reth1
member interface Reth2
member failover group 1
member failover group 2
node 1
bind chassis 1
priority 100
track 1 interface Blade1/2/0/1
track 3 interface Blade1/3/0/1
track 11 interface Route-Aggregation1
track 13 interface Route-Aggregation3
node 2
bind chassis 2
priority 50
track 2 interface Blade2/2/0/1
track 4 interface Blade2/3/0/1
track 12 interface Route-Aggregation2
track 14 interface Route-Aggregation4
Issues
Master/subordinate switchover fails for the IRF system through the redundancy group.
Solution
1. Check the track information for the redundancy group.
Track information is the only data source for the redundancy group to make decisions. Track configuration is of great importance for the redundancy group. If Track is incorrectly configured, the redundancy group might make wrong decisions.
a. For the redundancy group repeatedly activates members, check whether the associated track events are reported. In addition, check whether relationship of the interfaces is consistent with the nodes where the track entries are configured.
b. If no such problems exist, identify whether the track events match the interface status.
c. When a master/subordinate switchover occurs in the IRF system, identify whether the interfaces associated with the track event are in Positive status. If an interface is in Negative status, an anomaly exists.
d. If no such problems exist, identify whether the track entry status is consistent with that in the redundancy group.
# View the track entry status.
<sysname>dis track 5
Track ID: 5
State: Positive
Duration: 0 days 0 hours 0 minutes 6 seconds
Tracked object type: Interface
Notification delay: Positive 0, Negative 0 (in seconds)
Tracked object:
Interface: Route-Aggregation1
Protocol: None
# View the track entry status in the redundancy group.
<sysname>display redundancy group 1
Redundancy group 1 (ID 1):
Node ID Chassis Priority Status Track weight
1 Chassis1 100 Primary 255
2 Chassis2 50 Secondary 255
Preempt delay time remained : 0 min
Preempt delay timer setting : 1 min
Remaining hold-down time : 0 sec
Hold-down timer setting : 1 sec
Manual switchover request : No
Member interfaces:
Reth1
Reth2
Member failover groups:
1
2
Node 1:
Track info:
Track Status Reduced weight Interface
1 Positive 255 Blade1/2/0/1
3 Positive 255 Blade1/3/0/1
11 Positive 255 RAGG1
13 Positive 255 RAGG3
Node 2:
Track info:
Track Status Reduced weight Interface
2 Positive 255 Blade2/2/0/1
4 Positive 255 Blade2/3/0/1
12 Positive 255 RAGG2
14 Positive 255 RAGG4
If the information are not consistent, a track issue exists.
2. Verify that the weight processing for the redundancy is correct during IRF master/subordinate switchover
Each redundancy group node has a weight. The default value is 255. Each redundancy group node must be associated with at least one track entry. Each track entry corresponds to a weight increment value. When the track entry status becomes NotReady or Negative, the redundancy node substracts the associated weight increment from the current weight to obtain the new weight. When the track entry status becomes Positive, the redundancy node adds the associated weight increment to the current weight to obtain the new weight. When the weight is less than or equal to 0, the node is considered faulty and cannot operate correctly. A switchover or switchback operation is performed for the redundancy group.
# View the redundancy information as follows:
<sysname>display redundancy group 1
Redundancy group 1 (ID 1):
Node ID Chassis Priority Status Track weight
1 Chassis1 100 Secondary 0
2 Chassis2 50 Primary 255
Preempt delay time remained : 0 min
Preempt delay timer setting : 1 min
Remaining hold-down time : 0 sec
Hold-down timer setting : 1 sec
Manual switchover request : No
Member interfaces:
Reth1
Member failover groups:
1
2
Node 1:
Track info:
Track Status Reduced weight Interface
1 Positive 255 Blade1/2/0/1
3 Positive 255 Blade1/3/0/1
11 Negative(Faulty) 255 RAGG11
13 Positive 255 RAGG3
Node 2:
Track info:
Track Status Reduced weight Interface
2 Positive 255 Blade2/2/0/1
4 Positive 255 Blade2/3/0/1
12 Positive 255 RAGG2
14 Positive 255 RAGG4
3. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display redundancy group |
Displays redundancy group information. |
|
display track |
Displays track entry information. |
|
display reth interface Reth |
Displays Reth interface status. |
|
display interface |
Displays interface information. |
Troubleshooting policy NAT
Failure to access the external network from internal users
Symptom
PC A in the internal network cannot access PC B in the external network through the gateway device.
Figure 7 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy1.
¡ Source zone—Select the zone to which the interface connected to PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the zone to which the interface connected to PC B belongs as the destination zone. In this example, the destination zone is Untrust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 192.168.1.1.
¡ Destination IPv4 address—Specify the IP address of PC B as the destination IP. In this example, the address is 10.0.0.2.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Policy page.
3. Click Create.
4. Configure policy parameters as needed:
¡ Rule name—Specify the rule name. In this example, the name is policy1.
¡ Change Mode—Select source address translation as the change mode.
¡ Source zone—Select the zone to which the interface connected to PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the zone to which the interface connected to PC B belongs as the destination zone. In this example, the destination zone is Untrust.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 192.168.1.1.
¡ Destination IPv4 address—Specify the IP address of PC B as the destination IP. In this example, the address is 10.0.0.2.
¡ Translation method—Select Dynamic IP+port as the translation method.
¡ Address—Select a NAT address type for source address translation. In this example, the address type is NAT address group.
¡ Source address after NAT—Select a public NAT address group for source address translation.
5. Click OK.
Source address translation failure
Symptom
Two PCs are connected through the device configured with source address translation. However, PC A in the internal network cannot access PC B in the external network.
Figure 8 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy2.
¡ Source zone—Select the zone to which the interface connected to PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the zone to which the interface connected to PC B belongs as the destination zone. In this example, the destination zone is Untrust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 192.168.1.1.
¡ Destination IPv4 address—Specify the IP address of PC B as the destination IP. In this example, the address is 10.0.0.2.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Policy page.
3. Click the Edit icon to modify the source address translation rule.
4. In the Modify NAT policy dialog box that opens, check if addresses of the following parameters are within the 10.0.0.1/24 network:
¡ Source IP addresses after NAT.
¡ Network addresses for address translation.
¡ Address objects in the address group for address translation.
¡ Addresses in the NAT Address group for address translation.
5. If any, modify related configurations to verify return packets can be transmitted to interface GE 1/0/2 on the device.
6. Click OK.
Destination address translation failure
Symptom
Two PCs are connected through the device configured with destination address translation. However, PC B in the external network cannot access PC A in the internal network.
Figure 9 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy3.
¡ Source zone—Select the zone to which the interface connected to PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select the zone to which the interface connected to PC A belongs as the destination zone. In this example, the destination zone is Trust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of PC A as the destination IP. In this example, the address is 192.168.1.1.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Policy page.
3. Click the Edit icon.
4. Modify the destination address translation rule.
5. Click OK.
Destination address translation failure (source address translation in unification with destination address translation)
Symptom
PC B and PC C are connected through the device configured with the NAT Server feature. However, PC B cannot access PC C by using public address 10.0.0.100 and destination port 80.
Figure 10 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy4.
¡ Source zone—Select the zone to which the interface connected to PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select the zone to which the interface connected to PC C belongs as the destination zone. In this example, the destination zone is DMZ.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of PC C as the destination IP. In this example, the address is 192.168.2.1.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Policy page.
3. If a rule displays Dynamic IP+port in the Translation method column, click the Edit icon.
4. In the dialog box that opens, remove port number 80 from the port range in the NAT address group.
5. Click OK.
IPsec configuration failure (NAT in unification with IPsec)
Symptom
Two PCs are connected through the device configured with both IPsec and source address translation. However, when PC A sends a packet to PC B, the device cannot perform IPsec protection for the packet after source address translation.
Figure 11 Network diagram
Solution
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Network > VPN > IPsec > IPsec Policies page.
3. To modify a single IPsec policy, click the Edit icon.
4. In the Data flow filter rule area, change the source and destination addresses to addresses after NAT for the flows to be protected by IPsec.
5. Click OK.
Failure to access the gateway device configured with policy-based NAT from internal users
Symptom
PC A in the internal network cannot access the device.
Figure 12 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy5.
¡ Source zone—Select the zone to which the interface connecting PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select Local as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 192.168.1.1.
¡ Destination IPv4 address—Specify the IP address of the interface connected to the internal network as the destination IP. In this example, the address is 192.168.1.2.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Policy page.
3. If a rule displays Any in the Source security zone column, click the Edit icon.
4. In the dialog box that opens, modify the following packet match parameters for the destination address translation:
¡ Dst zone—Specify a destination security zone. The value for the parameter cannot be Local.
¡ Source IP—Specify a source IPv4 address. The value for the parameter cannot be 192.168.1.1.
¡ Destination IP—Specify a destination IPv4 address. The value for the parameter cannot be 192.168.1.2.
5. Click OK.
Failure to access the gateway device configured with source address translation from external users
Symptom
Two PCs are connected through the gateway device configured with source address translation. However, PC B cannot access the device.
Figure 13 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy6.
¡ Source zone—Select the zone to which the interface connected to PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select Local as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of the interface connected to the external network as the destination IP. In this example, the address is 10.0.0.1.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Policy page.
3. If a rule of source address translation displays Dynamic IP in the Translation method column, click the Edit icon to edit the rule.
4. If the address object group or NAT address group for source address translation contains the address of the interface connected to the external network, remove the address from the groups.
5. Click OK.
Failure to access the gateway device configured with destination address translation from external users
Symptom
Two PCs are connected through the gateway device configured with destination address translation. However, the external PC B cannot access the device.
Figure 14 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy7.
¡ Source zone—Select the zone to which the interface connected to PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select Local as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of the interface connected to the external network on Device as the destination IP. In this example, the address is 10.0.0.1.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Log in to the Web interface of the device.
3. Access the Policies > NAT > NAT Policy page.
4. If a rule entry of destination address translation displays Multiple to one address translation in the Destination address translation method column, click the Edit icon.
5. If the destination address match condition contains the address of the interface connected to the external network on Device, check the service match condition.
6. If the condition contains the service that PC B uses to access Device, perform the following solutions based on the actual situation:
¡ Modify the service that PC B uses to access Device.
¡ Remove the service from the service match condition to ensure the NAT module does not perform destination address translation on traffic containing the service.
7. Click OK.
Troubleshooting interface NAT
Failure to access the external network from internal users
Symptom
PC A in the internal network cannot access PC B in the external network through the gateway device.
Figure 15 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy1.
¡ Source zone—Select the zone to which the interface connecting PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the zone to which the interface connecting PC B belongs as the destination zone. In this example, the destination zone is Untrust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 192.168.1.1.
¡ Destination IPv4 address—Specify the IP address of PC B as the destination IP. In this example, the address is 10.0.0.2.
5. Click OK.
Solution (interface NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > Dynamic NAT page.
3. Click the Outbound Dynamic NAT (ACL-Based) tab.
4. Click Create.
5. Configure policy parameters as needed:
¡ Interface—Specify an interface. In this example, the interface is GE1/0/2.
¡ ACL—Specify an ACL to define the source IP addresses of outgoing packets to be translated by the NAT module.
¡ Source address after NAT—Specify a NAT address group. In this example, IP addresses are public addresses used for source address translation.
¡ Translation mode—Select PAT as the translation mode.
6. Click OK.
Source address translation failure
Symptom
Two PCs are connected through the device configured with source address translation. However, PC A in the internal network cannot access PC B in the external network.
Figure 16 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy2.
¡ Source zone—Select the zone to which the interface connected to PC A belongs as the source zone. In this example, the source zone is Trust.
¡ Destination zone—Select the zone to which the interface connected to PC B belongs as the destination zone. In this example, the destination zone is Untrust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC A as the source IP. In this example, the address is 192.168.1.1.
¡ Destination IPv4 address—Specify the IP address of PC B as the destination IP. In this example, the address is 10.0.0.2.
5. Click OK.
Solution (interface NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Click the Edit icon to modify the source address translation rule.
3. In the dialog box that opens, check if addresses of the following parameters are within the 10.0.0.1/24 network:
¡ Source addresses after NAT.
¡ Network addresses for address translation.
¡ Address objects in the address object group for address translation.
¡ Addresses in the NAT address group for address translation.
4. If any, modify related configurations to verify return packets can be transmitted to interface GE 1/0/2 on the device.
5. Click OK.
Destination address translation failure
Symptom
Two PCs are connected through the device configured with destination address translation. However, PC B in the external network cannot access PC A in the internal network.
Figure 17 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy3.
¡ Source zone—Select the zone to which the interface connected to PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select the zone to which the interface connected to PC A belongs as the destination zone. In this example, the destination zone is Trust.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of PC A as the destination IP. In this example, the address is 192.168.1.1.
5. Click OK.
Solution (interface NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Servers > Policy Configuration page.
3. Check whether the public port of the NAT Server is in actual use.
4. If not, change and verify the public port is in actual use for the NAT Server rule.
5. Click OK.
Destination address translation failure (source address translation in unification with destination address translation)
Symptom
PC B and PC C are connected through the device configured with the NAT Server feature. However, PC B cannot access PC C by using public address 10.0.0.100 and destination port 80.
Figure 18 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy4.
¡ Source zone—Select the zone to which the interface connected to PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select the zone to which the interface connected to PC C belongs as the destination zone. In this example, the destination zone is DMZ.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of PC C as the destination IP. In this example, the address is 192.168.2.1.
5. Click OK.
Solution (interface NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > Dynamic NAT page.
3. Click the Outbound Dynamic NAT (Group-Based) tab to view rule details.
4. If the action is PAT, click the Edit icon to remove port number 80 from the port range in the NAT address group.
5. Click OK.
6. Click the Outbound Dynamic NAT (ACL-Based) tab to view rule details.
7. If the translation mode is PAT, click the Edit icon to remove port number 80 from the port range in the NAT address group.
8. Click OK.
IPsec configuration failure (NAT in unification with IPsec)
Symptom
Two PCs are connected through the device with both IPsec and source address translation are configured correctly. However, when PC A sends a packet to PC B, the device cannot perform IPsec protection for the packet after source address translation.
Figure 19 Network diagram
Solution
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Network > VPN > IPsec > IPsec Policies page.
3. Click the Edit icon to edit the IPsec policy configurations.
4. In the Data flow filter rule area, use addresses after NAT as the source and destination addresses to match packets to be protected by IPsec.
5. Click OK.
Failure to access the gateway device configured with source address translation from external users
Symptom
Two PCs are connected through the gateway Device configured with source address translation. However, PC B cannot access the device.
Figure 20 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy5.
¡ Source zone—Select the zone to which the interface connecting PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select Local as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of the interface connected to the external network as the destination IP. In this example, the address is 10.0.0.1.
5. Click OK.
Solution (policy-based NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > Dynamic NAT page.
3. Click the Outbound Dynamic NAT (Group-Based) tab to check if the Action column is NO-PAT.
4. If the action is NO-PAT, click the Edit icon to remove IP address 10.0.0.1 from the NAT address group for packet match.
5. Click OK.
6. Click the Outbound Dynamic NAT (ACL-Based) tab to check if the Translation method column is NO-PAT.
7. If the translation mode is NO-PAT, click the Edit icon to open the NAT Dynamic NAT Rule dialog box:
¡ If IP addresses for address translation belong to an address group, verify the NAT address group does not contain 10.0.0.1.
¡ If the translation mode is Easy IP, the specified interface for address translation cannot be GE1/0/2.
8. Click OK.
Failure to access the gateway device configured with destination address translation from external users
Symptom
Two PCs are connected through the gateway device configured with destination address translation. However, the external PC B cannot access the device.
Figure 21 Network diagram
Solution (security policy)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > Security Policies > Security Policies page.
3. Click Create and then click Create a policy.
4. Configure policy parameters as needed:
¡ Name—Specify the policy name. In this example, the name is secpolicy6.
¡ Source zone—Select the zone to which the interface connecting PC B belongs as the source zone. In this example, the source zone is Untrust.
¡ Destination zone—Select Local as the destination zone.
¡ Action—Select Permit as the action.
¡ Source IPv4 address—Specify the IP address of PC B as the source IP. In this example, the address is 10.0.0.2.
¡ Destination IPv4 address—Specify the IP address of the interface connected to the external network on Device as the destination IP. In this example, the address is 10.0.0.1.
5. Click OK.
Solution (interface NAT)
To resolve the issue:
1. Log in to the Web interface of the device.
2. Access the Policies > NAT > NAT Servers > Policy Configuration page.
3. Check if the IP address 10.0.0.1 is occupied by a NAT Server rule.
4. If it is displayed in the Public IP address column, click the Edit icon for the rule.
5. In the Edit NAT Server Rule dialog box that opens, check if the public port occupies the port for PC B to access Device.
6. If the port is occupied, choose one of the following solutions:
¡ Modify the protocol or destination port for traffic from PC B to the device.
¡ Modify the ACL-based packet match rule to prevent the traffic to be processed by destination address translation.
7. Click OK.
Dynamic NAT failure
Symptom
Dynamic NAT fails or the translated packets cannot be forwarded correctly. PC A in the internal network cannot access PC B in the external network through the gateway device.
Solution
To resolve the issue:
1. Verify that NAT is configured correctly. This section uses outbound NAT as an example.
[H3C] display nat outbound
NAT outbound information:
There are 1 NAT outbound rules.
Interface: Route-Aggregation12
ACL: --- Address group: 257 Port-preserved: N
NO-PAT: N Reversible: N
2. Use the debugging nat packet command to enable debugging of NAT packets and verify that packets can be translated correctly. Example debugging information is as follows:
*May 13 09:58:48:083 2017 H3C NAT/7/COMMON: -slot =1;
PACKET: (Route-Aggregation12-in) Protocol: TCP
4.4.4.6: 21 - 4.4.5.11:11000(VPN: 0) ------>
4.4.4.6: 21 - 192.168.1.2:13249(VPN: 0)
3. Verify that session information is correct.
<sysname> display session table ipv4 verbose
Initiator:
Source IP/port: 192.168.1.2/13790
Destination IP/port: 4.4.4.6/21
DS-Lite tunnel peer: -
VPN instance/VLAN ID/VLL ID: -/-/-
Protocol: TCP(6)
Responder:
Source IP/port: 4.4.4.6/21
Destination IP/port: 4.4.4.27/1060
DS-Lite tunnel peer: -
VPN instance/VLAN ID/VLL ID: -/-/-
Protocol: TCP(6)
State: TCP_ESTABLISHED
Application: FTP
Start time: 2013-12-15 10:49:00 TTL: 3592s
Interface(in) : Route-Aggregation11
Interface(out): Route-Aggregation12
Zone(in) : Trust
Zone(out): menglei
Initiator->Responder: 3 packets 128 bytes
Responder->Initiator: 2 packets 130 bytes
4. Verify that the OpenFlow entries are consistent with the NAT session entries.
For dynamic NAT, the NAT entries will be deployed to each service module for traffic distribution.
[SYSNAME-probe] display system internal openflow instance inner flow-table
Flow entry rule 6 information:
cookie: 0x0, priority: 7301, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG1021
Ethernet type: 0x0800
IP Range: IPv4 destination address from 4.4.4.25 to 4.4.4.27
Instruction information:
Write actions:
Output interface: Blade2/4/0/1
Flow entry rule 7 information:
cookie: 0x0, priority: 7301, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG1021
Ethernet type: 0x0800
IP Range: IPv4 destination address from 4.4.4.28 to 4.4.4.30
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
5. If the issue persists, contact H3C Support.
Static NAT444 failure
Symptom
Figure 22 Network diagram
Network configuration
Configure static NAT444 on the device to allow PC1 to access PC2. The public address pool contains IP addresses 4.4.5.11 to 4.4.5.13. The device has two firewall modules.
Device configuration
# Configure NAT444 port block group.
nat port-block-group 256
local-ip-address 192.168.1.2 192.168.1.11 vpn-instance vpn11
global-ip-pool 4.4.5.11 4.4.5.12
block-size 1000
port-range 10000 19000
# Configure the input interface.
interface Route-Aggregation1023
ip binding vpn-instance vpn11
ip address 192.168.1.254 24
# Configure the output interface.
interface Route-Aggregation1021
ip address 4.4.4.254 255.255.255.0
nat outbound port-block-group 256
# Configure routes from vpn11 to the public network. (Details not shown.)
Issues
NAT444 fails, or the NAT-translated packets or return packets cannot be forwarded correctly.
Solution
To resolve the issue:
1. Verify that the port block group is configured correctly.
<sysname> display nat port-block-group 256
Port block group 256:
Port range: 10000-19000
Block size: 1000
Local IP address information:
Start address End address VPN instance
192.168.1.2 192.168.1.11 vpn11
Global IP pool information:
Start address End address
4.4.5.11 4.4.5.12
2. Verify that the number of port blocks and public addresses meet the private address requirements.
The port block for each private network has 1000 ports.
The private address range 192.168.1.2 to 192.168.1.11 requires 10 port blocks.
The port range is 10000 to 19000. Each public address can provide nine port blocks.
Ten private addresses require two public addresses. The settings meet the requirements.
3. Use the debugging nat packet command to enable debugging for NAT packets and verify that packets can be translated correctly.
4. Use the display session table ipv4 verbose command to verify that session information is correct.
5. Verify that the flow entries are deployed correctly.
[H3C-probe] display system internal openflow instance inner flow-table
Flow entry rule 24 information:
cookie: 0x0, priority: 7521, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG1021
Ethernet type: 0x0800
IP Range: IPv4 destination address from 4.4.5.11 to 4.4.5.12
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Flow entry rule 25 information:
cookie: 0x0, priority: 7500, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 source address from 192.168.1.2 to 192.168.1.11
VRF index: 16
[sysname] display ip vpn-instance instance-name
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
Flow entry rule 26 information:
cookie: 0x0, priority: 7501, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP Range: IPv4 destination address from 192.168.1.2 to 192.168.1.11
VRF index: 16
Instruction information:
Write actions:
Output interface: Blade2/10/0/1
A total of three flow entries are deployed. For static NAT444, all flow entries are deployed to the main security engine.
Use the display blade-controller-team default command to identify the main security engine.
<M9KS-2>display blade-controller-team Default
ID: 1 Name: Default
Chassis Slot CPU Status LBGroupID
2 3 1 Normal 1
* 2 4 1 Normal 1
* : Primary blade controller of the team.
¡ IP Range:IPv4 destination address from 4.4.5.11 to 4.4.5.11—Indicates the security engine to which the traffic (addresses after NAT translation) from PC2 to PC1 is sent.
¡ IP Range:IPv4 source address from 192.168.1.2 to 192.168.1.2—Indicates the security engine to which the traffic from PC1 to PC2 is sent.
¡ IP Range:IPv4 destination address from 192.168.1.2 to 192.168.1.2—Indicates the security engine to which the traffic from another PC (on the same network side as PC1) to PC1 is sent.
6. Verify that the session entries and flow entries are consistent.
7. If the issue persists, contact H3C Support.
NAT fails but the outbound interface can be pinged successfully from the external network
Symptom
The device acts as the gateway at the network egress. NAT fails on the gateway and the internal and external users cannot reach each other, but the outbound interface can be pinged successfully from the external network.
Solution
To resolve the issue:
1. Verify that the NAT address group is in the same subnet as the interface. If they are in different subnets, make sure a route to the NAT address group is configured on the peer end.
2. If the NAT address group or the NAT server address is in the same subnet as the interface, verify that the address group or the NAT server can send gratuitous ARP packets and the peer end has learned to correct MAC addresses. You can verify gratuitous ARP packet sending on a directly connected device.
A device cannot detect link disassociation of a non-directly connected device and cannot update the corresponding ARP entry. Gratuitous ARP enables a device to send ARP packets carrying the address of the sending device as the target address upon device association. Devices that receive the gratuitous ARP packets update their ARP entries. The address pool might fail to update its addresses in time.
3. Enable debugging or capture packets on the device, and verify that ping packets can be forwarded correctly (both ping requests and responses can be detected).
4. Ping the NAT address group or NAT server from the peer device continuously. Enable ARP debugging and verify that ARP packets can be received correctly.
5. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display nat outbound |
Displays outbound dynamic NAT configuration. |
|
display nat server |
Displays NAT server mappings. |
|
display session |
Displays session version information. |
|
save |
Saves the running configuration to a configuration file. |
Troubleshooting AFT
IPv6 access to IPv4 network fails
Symptom
This section uses IPv6-to-IPv4 source address dynamic translation and IPv4-to-IPv6 source address static translation as an example.
To allow PC1 to access PC2 in the IPv4 network, the following AFT policies are configured on the device:
· An IPv4-to-IPv6 source address static mapping to map destination IPv4 address 1.1.1.1 to IPv6 address 23::1.
· An IPv6-to-IPv4 source address dynamic translation policy to translate the source addresses in IPv6 packets to IPv4 address 30.30.40.100.
However, AFT fails or the translated packets cannot be forwarded correctly.
Figure 23 Network diagram
Settings on the device
acl ipv6 number 2000
rule 0 permit source 1:1::1/128
#
aft address-group 0
address 30.30.40.100 30.30.40.100
#
aft v6tov4 source acl ipv6 number 2000 address-group 0
#
aft v4tov6 source 1.1.1.1 23::1
#
interface Route-Aggregation10.900
aft enable
interface Route-Aggregation10.901
aft enable
Solution
To resolve the issue:
1. Verify that AFT is configured correctly and AFT is enabled on both the input interface and output interface on the device.
[sysname]dis aft configuration
aft address-group 0
address 30.30.40.100 30.30.40.100
aft v6tov4 source acl ipv6 number 2000 address-group 0
aft v4tov6 source 1.1.1.1 23::1
interface Route-Aggregation10.900
aft enable
interface Route-Aggregation10.901
aft enable
AFT ALG:
DNS : Enabled
FTP : Enabled
HTTP : Enabled
ICMP-ERROR : Enabled
RTSP : Enabled
SIP : Enabled
2. Use the debugging aft packet ip command to enable debugging of AFT packets and verify that packets can be translated correctly. If the following information is displayed, AFT packets are translated correctly.
<sysname>debugging aft packet ip
Dec 16 15:08:22:697 2020 H3C AFT/7/COMMON: -Slot=6.1;
PACKET: (Route-Aggregation10.900) Protocol: UDP
1.1.1.1/69 - 30.30.40.100/1128(VPN:0) ------>
23::1/69 – 1:1::1/35017(VPN:0)
Or
<sysname>debugging aft packet ipv6
Dec 16 15:09:13:696 2020 H3C AFT/7/COMMON: -Slot=6.1;
PACKET: (Route-Aggregation10.901) Protocol: UDP
1:1::1/6677 - 23::1/5060(VPN:0) ------>
30.30.40.100/1149 - 1.1.1.1/5060(VPN:0)
3. Verify that the flow tables are deployed normally.
[H3C-probe]dis system internal openflow instance inner-redirect flow-table
Flow entry 3305 information:
cookie: 0x0, priority: 5045, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 900, mask: 0xfff
IP Range: IPv4 destination address from 30.30.40.100 to 30.30.40.100
Instruction information:
Write actions:
Group: 4026531857
Flow entry 3306 information:
cookie: 0x0, priority: 5045, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 4094, mask: 0xfff
IP Range: IPv4 destination address from 30.30.40.100 to 30.30.40.100
Instruction information:
Write actions:
Group: 4026531857
Flow entry 3307 information:
cookie: 0x0, priority: 5080, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
IPv4 source address: 1.1.1.1, mask: 255.255.255.255
Instruction information:
Write actions:
Group: 4026531865
Flow entry 3308 information:
cookie: 0x0, priority: 5085, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
IPv4 destination address: 1.1.1.1, mask: 255.255.255.255
Instruction information:
Write actions:
Group: 4026531865
Flow entry 3309 information:
cookie: 0x0, priority: 7085, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 900, mask: 0xfff
IPv6 destination address: 23::1
IPv6 destination address mask: FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
Instruction information:
Write actions:
Group: 4026531865
Flow entry 3310 information:
cookie: 0x0, priority: 7085, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 4094, mask: 0xfff
IPv6 destination address: 23::1
IPv6 destination address mask: FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
Instruction information:
Write actions:
Group: 4026531865
4. If the issue persists, contact Technical Support.
Troubleshooting IPsec and IKE
IPsec SAs established successfully but IPsec-protected traffic cannot be forwarded
Symptom
An IKE-based IPsec tunnel is established successfully between Device 1 and Device 2 to protect the traffic between PC 1 and PC 2, but the PCs cannot ping each other.
Figure 24 Network diagram
Settings on Device 1:
· The local address and remote address of the IPsec tunnel are 9.9.9.9 and 9.9.9.19, respectively.
· The ACL rule for IPsec:
rule 0 permit ip source 81.2.0.0 0.0.0.255 destination 82.2.0.0 0.0.0.255
Settings on Device 2:
· The local address and remote address of the IPsec tunnel are 9.9.9.19 and 9.9.9.9, respectively.
· The ACL rule for IPsec:
rule 0 permit ip source 82.2.0.0 0.0.0.255 destination 81.2.0.0 0.0.0.255
Solution
To resolve the issue:
1. Verify that the IKE SAs and IPsec SAs have been established on Device 1, and the status of the IPsec SAs and the OpenFlow entries deployed by IPsec is active.
# Display the IKE SAs on Device 1.
[sysname]dis ike sa
Connection-ID Remote Flag DOI
------------------------------------------------------------------
1 9.9.9.9 RD IPsec
Flags:
RD--READY RL--REPLACED FD-FADING RK-REKEY
# Display the IPsec SAs generated on Device 1.
[sysname]dis ipsec sa
-------------------------------
Interface: Ten-GigabitEthernet8/2/20
-------------------------------
-----------------------------
IPsec policy: ipsec
Sequence number: 1
Mode: ISAKMP
Flow table status: Active
-----------------------------
Tunnel id: 0
Encapsulation mode: tunnel
Perfect Forward Secrecy:
Inside VPN:
Extended Sequence Numbers enable: N
Traffic Flow Confidentiality enable: N
Path MTU: 1428
Tunnel:
local address: 9.9.9.19
remote address: 9.9.9.9
Flow:
sour addr: 152.2.0.0/255.255.0.0 port: 0 protocol: ip
dest addr: 151.1.0.0/255.255.0.0 port: 0 protocol: ip
[Inbound ESP SAs]
SPI: 42602698 (0x028a10ca)
Connection ID: 4294967296
Transform set: ESP-ENCRYPT-AES-CBC-128 ESP-AUTH-SHA1
SA idle time: 86400
SA duration (kilobytes/sec): 1843200/3600
SA remaining duration (kilobytes/sec): 1843199/3154
Max received sequence-number: 4
Anti-replay check enable: Y
Anti-replay window size: 64
UDP encapsulation used for NAT traversal: N
Status: Active
[Outbound ESP SAs]
SPI: 3182510800 (0xbdb142d0)
Connection ID: 4294967297
Transform set: ESP-ENCRYPT-AES-CBC-128 ESP-AUTH-SHA1
SA idle time: 86400
SA duration (kilobytes/sec): 1843200/3600
SA remaining duration (kilobytes/sec): 1843199/3154
Max sent sequence-number: 4
UDP encapsulation used for NAT traversal: N
Status: Active
2. Verify that the flow entries have been deployed by IPsec on the interface modules of Device 2.
If phase 1 and phase 2 negotiations are successful, IPsec deploys two OpenFlow entries:
¡ For encryption, upon receiving a clear text packet, IPsec deploys an entry in the form of an ACL rule indicating the source address and destination address of the flow.
¡ For decryption, upon receiving an encrypted packet, IPsec deploys an entry indicating the source and destination IP addresses of the tunnel, and the VRF index.
[h3c-probe]display system internal openflow instance inner-redirect flow-tab
le
Instance 4097 flow table information:
Flow entry 41 information:
cookie: 0x0, priority: 8102, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP protocol: 50
IPv4 source address: 9.9.9.19, mask: 255.255.255.255
IPv4 destination address: 9.9.9.9, mask: 255.255.255.255
VRF index: 0
Instruction information:
Write actions:
Group: 4026531873
Flow entry 42 information:
cookie: 0x0, priority: 8300, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
IPv4 source address: 151.1.0.0, mask: 255.255.0.0
IPv4 destination address: 152.2.0.0, mask: 255.255.0.0
Instruction information:
Write actions:
Group: 4026531873
3. Verify that the flow entries have been deployed by IPsec on the service modules of Device 2.
[h3c-probe]display system internal openflow instance inner flow-table
Instance 4096 flow table information:
Flow entry 21 information:
cookie: 0x0, priority: 8102, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP protocol: 50
IPv4 source address: 9.9.9.19, mask: 255.255.255.255
IPv4 destination address: 9.9.9.9, mask: 255.255.255.255
VRF index: 0
Instruction information:
Write actions:
Group: 4026531873
Flow entry 22 information:
cookie: 0x0, priority: 8300, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IPv4 source address: 151.1.0.0, mask: 255.255.0.0
IPv4 destination address: 152.2.0.0, mask: 255.255.0.0
Instruction information:
Write actions:
Group: 4026531873
4. Use the reset ipsec sa and reset ike sa commands to clear and re-establish IPsec SAs and IKE SAs.
5. If the issue persists, contact Technical Support.
IPsec exceptions occur when the master device in IRF fabric goes down
Symptom
Device 1 and Device 2 form an IRF fabric, where Device 1 is the master and Device 2 is the backup. An IKE-based IPsec tunnel is established successfully between the FW and the IRF fabric to protect the traffic between PC 1 and PC 2. Normally, the IPsec traffic is transmitted on Device 1. When Device 1 becomes down, the PCs cannot ping each other.
Figure 25 Network diagram
Solution
To resolve the issue:
1. Verify that the IKE SAs and IPsec SAs have been established on Device 2.
2. If the IKE SAs and IPsec SAs have not been established on Device 2, use the display system internal openflow instance command to verify that IPsec-related flow entries exist on Device 2.
For example, the following output shows that IPsec-related flow entries have not been cleared after the master device Device 1 is down. Therefore, no new SAs can be established.
[h3c-probe]display system internal openflow instance inner-redirect flow-table
Instance 4097 flow table information:
Flow entry 41 information:
cookie: 0x0, priority: 8102, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Ethernet type: 0x0800
IP protocol: 50
IPv4 source address: 9.9.9.19, mask: 255.255.255.255
IPv4 destination address: 9.9.9.9, mask: 255.255.255.255
VRF index: 0
Instruction information:
Write actions:
Group: 4026531873
Flow entry 42 information:
cookie: 0x0, priority: 8300, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
IPv4 source address: 151.1.0.0, mask: 255.255.0.0
IPv4 destination address: 152.2.0.0, mask: 255.255.0.0
Instruction information:
Write actions:
3. Perform a master-to-backup switchover, and verify that the current IPsec SAs can be re-established.
After master-to-backup switchover, if the service modules processing IPsec services on the master device or the master device becomes down, IPsec SAs can be re-established through the backup device. If the IPsec SAs are not established, go to the next step.
4. Use the reset ipsec sa and reset ike sa commands to clear and re-establish IPsec SAs and IKE SAs.
5. Use the debugging ipsec and debugging ike commands to debug exceptions.
6. If the issue persists, contact Technical Support.
Related commands
|
Command |
Description |
|
display ike sa |
Displays IKE SA information. |
|
display ipsec sa |
Displays IPsec SA information. |
|
display system internal openflow instance |
Displays instance and flow table information. |
|
reset ike sa |
Clears IKE SAs. |
|
reset ipsec sa |
Clears IPsec SAs. |
|
save |
Saves the running configuration to the specified file. |
IKE SAs established successfully but IPsec SAs cannot be established
Symptom
Establish an IKE-based IPsec tunnel between Device 1 and Device 2 to protect the traffic between PC 1 and PC 2. IKE SAs are established successfully, but IPsec SAs cannot be established.
Figure 26 Network diagram
Settings on Device 1:
· The local address and remote address of the IPsec tunnel are 81.2.0.1 and 14.5.1.1, respectively.
· The ACL rule for IPsec:
rule 0 permit ip source 81.2.0.0 0.0.0.255 destination 82.2.0.0 0.0.0.255
Settings on Device 2:
· The local address and remote address of the IPsec tunnel are 14.5.1.1 and 81.2.0.1, respectively.
· The ACL rule for IPsec:
rule 0 permit ip source 82.2.0.0 0.0.0.255 destination 81.2.0.0 0.0.0.255
Solution
To resolve the issue:
1. View the packet hit counts for the ACL rules, and verify that configuration for the ACL rules matches the traffic to be protected.
2. Verify that the FWs have consistent security protocol, encryption and authentication algorithms, and encapsulation mode settings.
3. Verify that the IPsec configuration is correct and complete. For example, make sure the remote address or IKE profile is correctly configured.
4. Use the reset ipsec sa and reset ike sa commands to clear and re-establish IPsec SAs and IKE SAs.
5. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display ike sa |
Displays IKE SA information. |
|
display ipsec sa |
Displays IPsec SA information. |
|
reset ike sa |
Clears IKE SAs. |
|
reset ipsec sa |
Clears IPsec SAs. |
|
display ipsec transform-set |
Displays IPsec transform set information. |
|
display ipsec policy |
Displays IPsec policy information. |
|
save |
Saves the running configuration to the specified file. |
IKE SAs cannot be established
Symptom
Establish an IKE-based IPsec tunnel between Device 1 and Device 2 to protect the traffic between PC 1 and PC 2. However, IKE SAs cannot be established.
Figure 27 Network diagram
Settings on Device 1:
· The local address and remote address of the IPsec tunnel are 81.2.0.1 and 14.5.1.1, respectively.
· The ACL rule for IPsec:
rule 0 permit ip source 81.2.0.0 0.0.0.255 destination 82.2.0.0 0.0.0.255
Settings on Device 2:
· The local address and remote address of the IPsec tunnel are 14.5.1.1 and 81.2.0.1, respectively.
· The ACL rule for IPsec:
rule 0 permit ip source 82.2.0.0 0.0.0.255 destination 81.2.0.0 0.0.0.255
Solution
To resolve the issue:
1. Verify that the FWs have consistent IKE proposal settings, mainly including the encryption and authentication algorithms, and the identify authentication mode.
2. Make sure the IKE identify authentication succeeds:
¡ For preshared key authentication, verify that the preshared keys configured on the FWs are the same.
¡ For certificate authentication, verify the following settings:
- The certificates on the FWs are within the validity period or are not withdrawn.
- The certificates have trusted CAs and are signed by the same CA.
- The FWs have matching keys in the certificates.
¡ Verify that no remote identity conflict exists. The conflict might be caused by identical remote identity configuration in different IKE profiles.
3. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display ike proposal |
Displays configuration information about all IKE proposals. |
|
display ike sa |
Displays IKE SA information. |
|
display ipsec sa |
Displays IPsec SA information. |
|
reset ike sa |
Clears IKE SAs. |
|
reset ipsec sa |
Clears IPsec SAs. |
|
save |
Saves the running configuration to the specified file. |
IPsec smart link does not detect link quality
Symptom
The branch accesses the headquarters through IPsec VPN. Configure IPsec smart link selection so the branch can establish an IPsec tunnel to the headquarters over link 1 or link 2, whichever has a better link quality.
· Device A first uses link 1 to establish the IPsec tunnel.
· When link 1 suffers high packet loss ratio or delay, Device A automatically switches traffic to the IPsec tunnel established based on link 2.
However, the IPsec smart link policy cannot function as expected. It does not detect the link qualities for smart link switchover.
Figure 28 Network diagram
Primary settings on Device A
# Configure interface IP addresses and gateway IP addresses.
<DeviceA> system-view
[DeviceA] interface gigabitethernet 1/0/1
[DeviceA-GigabitEthernet1/0/1] ip address 1.1.1.1 24
[DeviceA-GigabitEthernet1/0/1] gateway 1.1.1.3
[DeviceA-GigabitEthernet1/0/1] quit
[DeviceA] interface gigabitethernet 1/0/2
[DeviceA-GigabitEthernet1/0/2] ip address 2.2.2.2 24
[DeviceA-GigabitEthernet1/0/2] gateway 2.2.2.3
[DeviceA-GigabitEthernet1/0/2] quit
# Configure an IPsec smart link policy named policy1 and configure link 1 and link 2 in the policy.
[DeviceA] ipsec smart-link policy policy1
[DeviceA-ipsec-smart-link-policy-policy1] link 1 interface gigabitethernet 1/0/1 remote 3.3.3.3
[DeviceA-ipsec-smart-link-policy-policy1] link 2 interface gigabitethernet 1/0/2 remote 3.3.3.3
# Set the maximum number of link switchover cycles to 4.
[DeviceA-ipsec-smart-link-policy-policy1] link-switch cycles 4
# Enable IPsec smart link selection in the policy.
[DeviceA-ipsec-smart-link-policy-policy1] smart-link enable
[DeviceA-ipsec-smart-link-policy-policy1] quit
Primary settings on Device B
# Configure the interface IP address.
<DeviceB> system-view
[DeviceB] interface gigabitethernet 1/0/1
[DeviceB-GigabitEthernet1/0/1] ip address 3.3.3.3 24
[DeviceB-GigabitEthernet1/0/1] quit
# Configure an IPv4 ACL to identify data flows to be protected by IPsec.
[DeviceB] acl advanced 3000
[DeviceB-acl-ipv4-adv-3000] rule permit ip source 10.1.2.0 0.0.0.255 destination 10.1.1.0 0.0.0.255
[DeviceB-acl-ipv4-adv-3000] rule permit ip source 3.3.3.0 0.0.0.255 destination 1.1.1.0 0.0.0.255
[DeviceB-acl-ipv4-adv-3000] rule permit ip source 3.3.3.0 0.0.0.255 destination 2.2.2.0 0.0.0.255
[DeviceB-acl-ipv4-adv-3000] quit
# Configure static routes for Device B to reach the subnets connected to Device A. This example uses 3.3.3.1 as the next hop address of the routes.
[DeviceB] ip route-static 10.1.1.0 255.255.255.0 gigabitethernet 1/0/1 3.3.3.1
[DeviceB] ip route-static 1.1.1.0 255.255.255.0 gigabitethernet 1/0/1 3.3.3.1
[DeviceB] ip route-static 2.2.2.0 255.255.255.0 gigabitethernet 1/0/1 3.3.3.1
Solution
To resolve the issue:
1. Verify that the links are valid. For example, make sure the interfaces are assigned IP addresses and the interface state is up.
2. Verify that the smart link configuration is complete. For example, make sure the smart link policy is specified in the IPsec policy, and next hops for the routes are well configured.
3. Verify that IPsec policy settings are correct and complete.
4. Change the maximum number of link switchover cycles to a bigger value to ensure that the failure is not caused by the limit of link switchover cycles count.
5. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display ike sa |
Displays IKE SA information. |
|
display ipsec sa |
Displays IPsec SA information. |
|
reset ike sa |
Clears IKE SAs. |
|
reset ipsec sa |
Clears IPsec SAs. |
|
display ipsec smart-link policy |
Displays IPsec smart link policy information. |
|
display ipsec policy |
Displays IPsec policy information (to verify the IPsec policy configuration and view the specified smart link policy). |
|
display acl 3000 |
Displays information about dynamic ACL rules. |
|
save |
Saves the running configuration to the specified file. |
IPsec tunnel interface-based IPsec tunnel cannot be established
Symptom
Both the branch and the headquarters use fixed IP addresses to access the Internet.
To ensure that the IPsec configuration of the headquarters remains stable despite of changes of the branch subnet, IPsec tunnel interface-based IPsec is configured on Device A and Device B. This tunnel is used to protect all the traffic between the branch (10.1.1.0/24) and the headquarters (10.1.2.0/24).
However, the IPsec tunnel interface-based IPsec tunnel cannot be established as expected.
Figure 29 Network diagram
IPsec tunnel interface settings on Device A:
# Create IPsec tunnel interface Tunnel1.
[DeviceA] interface tunnel 1 mode ipsec
# Configure an IP address for the tunnel interface.
[DeviceA-Tunnel1] ip address 3.3.3.1 255.255.255.0
# Configure the IP address of GE 1/0/2 on Device A as the source address of the tunnel.
[DeviceA-Tunnel1] source 2.2.2.1
# Configure the IP address of GE 1/0/2 on Device B as the destination address of the tunnel.
[DeviceA-Tunnel1] destination 2.2.3.1
# Apply an IPsec profile to the tunnel interface.
[DeviceA-Tunnel1] tunnel protection ipsec profile abc
[DeviceA-Tunnel1] quit
# Configure a static route from Device A to Device B that goes through the tunnel interface.
[DeviceA] ip route-static 10.1.2.0 255.255.255.0 tunnel 1
IPsec tunnel interface settings on Device B:
# Create IPsec tunnel interface Tunnel1.
[DeviceB] interface tunnel 1 mode ipsec
# Configure an IP address for the tunnel interface.
[DeviceB-Tunnel1] ip address 3.3.3.2 255.255.255.0
# Configure the IP address of GE 1/0/2 on Device B as the source address of the tunnel.
[DeviceB-Tunnel1] source 2.2.3.1
# Configure the IP address of GE 1/0/2 on Device A as the destination address of the tunnel.
[DeviceB-Tunnel1] destination 2.2.2.1
# Apply an IPsec profile to the tunnel interface.
[DeviceB-Tunnel1] tunnel protection ipsec profile abc
[DeviceB-Tunnel1] quit
# Configure a static route from Device B to Device A that goes through the tunnel interface.
[DeviceB] ip route-static 10.1.1.0 255.255.255.0 tunnel 1
Solution
To resolve the issue:
1. Verify that the IPsec tunnel interface state is up. If the state is down, check the completeness of the tunnel configuration. The common check items include the source address, destination address (the command syntax might be misspelled), and tunnel interface IP address.
2. Verify that the tunnel source interface is up and the tunnel destination address is reachable.
3. Verify that IPsec and IKE basic settings are correct.
4. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display ike sa |
Displays IKE SA information. |
|
display ipsec sa |
Displays IPsec SA information. |
|
reset ike sa |
Clears IKE SAs. |
|
reset ipsec sa |
Clears IPsec SAs. |
|
display ip interface brief |
Displays interface state information. |
|
display interface Tunnel 1 |
Displays tunnel state information. |
|
save |
Saves the running configuration to the specified file. |
Troubleshooting load balancing
Traffic forwarding failure from client to server when the virtual and real servers are active in Layer 4 server load balancing
Symptom
As shown in Figure 30, physical servers Server A, Server B, and Server C provide FTP services, and are in descending order of hardware configuration. Configure server load balancing on the device to distribute user requests among the servers based on their hardware performance, and use health monitoring to monitor the reachability of the servers.
Configure virtual server vs on the network. The virtual server and real servers rs1, rs2, and rs3 are all active, but the client cannot access the virtual server address.
The following is the configuration procedure:
1. Configure a server farm.
# Create the ICMP NQA template t1.
#
nqa template icmp t1
#
Create the server farm sf, and specify the scheduling algorithm as weighted round robin and health monitoring method as t1.
#
server-farm sf
probe t1
#
2. Configure real servers.
# Create real server rs1 with IPv4 address 192.168.1.1 and weight 150, and add it to server farm sf.
#
real-server rs1
ip address 192.168.1.1
weight 150
server-farm sf
#
# Create real server rs2 with IPv4 address 192.168.1.2 and weight 120, and add it to server farm sf.
#
real-server rs2
ip address 192.168.1.2
weight 120
server-farm sf
#
# Create real server rs3 with IPv4 address 192.168.1.3 and weight 80, and add it to server farm sf.
#
real-server rs3
ip address 192.168.1.3
weight 80
server-farm sf
#
3. Configure a virtual server.
# Create TCP virtual server vs with VSIP 61.159.4.100, specify its default primary server farm sf, and enable the virtual server.
#
virtual-server vs type tcp
virtual ip address 61.159.4.100
default server-farm sf
service enable
#
Solution
1. View the virtual server statistics on the LB device to verify the reachability between the client and the LB device and the packet loss of the virtual server.
If the client cannot reach the LB device, the virtual server has no statistics. In this scenario, resolve the reachability issue and view the statistics again. If the statistics shows that packets are lost, enable debugging for load balancing or perform PCAP analysis on the client.
# Display statistics of virtual server vs.
[LB] display virtual-server statistics name vs
Slot 1:
Virtual server: vs
Total connections: 10
Active connections: 3
Max connections: 3
Connections per second: 0
Max connections per second: 1
Client input: 3210 bytes
Client output: 14074 bytes
Throughput: 0 bytes/s
Max throughput: 7554 bytes/s
Received packets: 1365
Sent packets: 2796
Dropped packets: 0
2. If the virtual server statistics is normal and has no packet loss, identify whether real servers in the server farm have packet loss.
If a real server has packet loss, enable debugging for load balancing or perform PCAP analysis on the responding server. From the results, verify the reachability between the real server and the LB device, and identify whether the service or port is enabled on the real server.
# Display statistics of real servers.
[LB] display real-server statistics name rs1
Slot 1:
Real server: rs1
Total connections: 5
Active connections: 1
Max connections: 1
Connections per second: 0
Max connections per second: 1
Server input: 307462 bytes
Server output: 27460 bytes
Throughput: 0 bytes/s
Max throughput: 316457 bytes/s
Received packets: 319
Sent packets: 236
Dropped packets: 0
Received requests: 0
Dropped requests: 0
Sent responses: 0
Dropped responses: 0
[LB]display real-server statistics name rs2
Slot 1:
Real server: rs2
Total connections: 2
Active connections: 1
Max connections: 1
Connections per second: 0
Max connections per second: 1
Server input: 870147 bytes
Server output: 45163 bytes
Throughput: 0 bytes/s
Max throughput: 580348 bytes/s
Received packets: 748
Sent packets: 511
Dropped packets: 0
Received requests: 0
Dropped requests: 0
Sent responses: 0
Dropped responses: 0
[LB]display real-server statistics name rs3
Slot 1:
Real server: rs3
Total connections: 2
Active connections: 1
Max connections: 1
Connections per second: 0
Max connections per second: 1
Server input: 870147 bytes
Server output: 45163 bytes
Throughput: 0 bytes/s
Max throughput: 580348 bytes/s
Received packets: 178
Sent packets: 311
Dropped packets: 0
Received requests: 0
Dropped requests: 0
Sent responses: 0
Dropped responses: 0
3. If no issues are identified from the statistics, enable debugging for load balancing and locate the failure from debugging information.
4. If the issue persists, contact H3C Support.
High CPU usage and memory usage
Symptom
Packet loss occurs on the virtual server, NQA operations fail, or concurrent connection count is low and new connections fail.
Solution
1. Display the real server state.
High CPU usage might cause failed NQA operations or packet loss on the virtual server. High memory usage causes new requests to be discarded.
2. If the issue persist, contact H3C Support.
Uneven load balancing
Symptom
Load balancing is uneven.
Solution
1. View real server statistics. Use the round robin algorithm for load balancing.
2. Use the least connection or random algorithm. The LB card has multiple CPU cores. Load balancing is performed per core. For this reason, connections might be distributed unevenly across real servers.
3. If the source IP address hash algorithm is used, make sure the number of source IP addresses is sufficient.
4. Configure an LB policy to achieve more granular traffic classification. Adjust the server state based on service requirements and actual network environment.
Related commands
|
Command |
Description |
|
debugging lb all |
Enables all debugging functions for load balancing. |
|
debugging lb error |
Enables load balancing error debugging. |
|
debugging lb event |
Enables load balancing event debugging. |
|
debugging lb fsm |
Enables load balancing state machine debugging. |
|
debugging lb packet |
Enables load balancing packet debugging. |
|
display real-server statistics [ name real-server-name ] |
Displays real server statistics. |
|
display virtual-server statistics [ name virtual-server-name ] |
Displays virtual server statistics. |
|
reset real-server statistics [ real-server-name ] |
Clears real server statistics. |
|
reset virtual-server statistics [ virtual-server-name ] |
Clears virtual server statistics. |
Troubleshooting system management
High CPU usage
Symptom
The CPU usage of the device remains over 60%, and command executions are very slow.
<sysname> display cpu-usage
Slot 1 CPU 0 CPU usage:
13% in last 5 seconds
13% in last 1 minute
13% in last 5 minutes
The display cpu-usage history command displays the CPU usage in the most recent 60 minutes.
<sysname> display cpu-usage history
100%|
95%|
90%|
85%|
80%|
75%|
70%|
65%|
60%|
55%|
50%|
45%|
40%|
35%|
30%|
25%|
20%|
15%|
10%|
5%| #
------------------------------------------------------------
10 20 30 40 50 60 (minutes)
cpu-usage (CPU 0) last 60 minutes (SYSTEM)
Solution
To resolve the issue:
1. Verify whether too many routing policies have been configured.
Use the display route-policy command to view the configured routing policies and determine whether too many routing policies have been configured leading to high CPU usage.
<sysname> display route-policy
Route-policy: policy1
permit : 1
if-match cost 10
continue: next node 11
apply comm-list a delete
2. Verify whether a traffic loop has occurred.
When a traffic loop occurs, the network flaps and a large number of protocol packets are sent to the CPU for processing, which might result in high CPU usage. A traffic loop might cause broadcasting. Many ports of the device have to process a large amount of traffic, causing the port utilization rate to reach 90% or more.
<sysname> display interface GigabitEthernet1/0/2
GigabitEthernet1/0/2 current state: UP
Line protocol current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 0000-e80d-c000
Description: GigabitEthernet2/6/0/1 Interface
Loopback is not set
Media type is optical fiber, Port hardware type is 1000_BASE_SX_SFP
1000Mbps-speed mode, full-duplex mode
……
Last clearing of counters: Never
Peak value of input: 123241940 bytes/sec, at 2013-06-27 14:33:15
Peak value of output: 80 bytes/sec, at 2013-06-27 14:13:00
Last 300 second input: 26560 packets/sec 123241940 bytes/sec 99%
Last 300 second output: 0 packets/sec 80 bytes/sec 0%
……
If a traffic loop occurs, perform the following steps.
a. Verify whether the link connection and port configuration are correct.
b. Verify whether STP is enabled on the connected switch, and whether the configuration is correct.
c. Verify whether the routing configuration is correct and whether a routing loop exists.
3. Verify whether the packets are fast forwarded.
Execute the display ip fast-forwarding cache command to view whether the forwarding entry for the packets exist in the output. If the entry does not exist, the packets are not fast forwarded.
<sysname> display ip fast-forwarding cache
Total number of fast-forwarding entries: 78
SIP SPort DIP DPort Pro Input_If Output_If Flg
40.1.20.2 65535 30.1.2.2 1024 6 Reth4 Reth3 1
192.168.96.40 53342 192.168.205.33 23 6 GE1/0/0 N/A 1
30.1.2.2 1024 40.1.20.2 65535 6 Reth3 Reth4 1
192.168.205.33 23 192.168.96.52 60824 6 InLoop0 GE1/0/0 1
120.0.0.1 1701 120.0.0.2 1701 17 InLoop0 GE1/0/2.120 1
40.1.20.2 65529 30.1.2.2 1024 6 Reth4 Reth3 1
130.2.1.115 1701 130.2.1.1 1701 17 Reth4 N/A 1
30.1.2.2 1024 40.1.20.2 65533 6 Reth3 Reth4 1
40.1.20.2 65526 30.1.2.2 1024 6 Reth4 Reth3 1
50.1.1.2 1024 60.1.1.2 1024 6 Reth1 Tun1 1
192.168.205.33 37932 192.168.100.53 0 1 InLoop0 GE1/0/0 1
30.1.2.2 1024 40.1.20.2 65529 6 Reth3 Reth4 1
30.1.2.2 1024 40.1.20.2 65527 6 Reth3 Reth4 1
60.1.1.2 1024 50.1.1.2 1024 6 Tun1 Reth1 1
40.1.20.2 65532 30.1.2.2 1024 6 Reth4 Reth3 1
You can also enter an IP address in the command to verify whether the packets that use the IP address as the source or destination IP address are fast forwarded.
<sysname> display ip fast-forwarding cache 12.1.1.1
Total number of fast-forwarding entries: 2
SIP SPort DIP DPort Pro Input_If Output_If Flg
12.1.1.2 49216 12.1.1.1 3784 17 InLoop0 N/A 1
12.1.1.1 3784 12.1.1.2 49216 17 RAGG5.3101 InLoop0 1
If the issue persists, execute the display cpu-usage command and provide the command output together with other related information to technical support for analysis.
4. Verify whether object policy or ACL acceleration is enabled.
#
object-policy ip EXTERNAL-Local
rule 0 pass vrf external_vpn
rule 1 pass vrf 7tgaklptgb9o19babgnm3kbst8
accelerate
#
An acceleration-disabled object policy or ACL that has more than 50 rules might cause high CPU usage. You can use the display object-policy accelerate summary ip and display acl accelerate summary commands to view which object policies and ACLs are enabled with acceleration.
5. Verify whether the IP address object group for an object policy is configured with excluded IP addresses or a discontinuous wildcard mask.
If an excluded IP address or a discontinuous wildcard mask is configured for the IP address object group, object policy acceleration will fail, causing high CPU usage. You must remove related configurations.
6. Verify whether the NAT444 port block resources are sufficient.
On a network configured with NAT444, bursts of traffic (the source port of the packets changes frequently while the source and destination IP addresses and destination port numbers remain unchanged) might cause exhaustion of NAT444 port resources.
Execute the display system internal nat statistics chassis X slot X cpu 1 | in failed command in probe view to view whether the count for the "NAT444 failed to translate port" field or similar increases significantly.
If the count is increasing significantly, use the display nat port-block static c 1 s X c 1 command to identify which address' s mapping occupies a large amount of port resources, and examine the configuration of NAT address group where the address belongs to determine whether the occupied port resource has reached the upper limit.
If the upper port resource limit is reached, change the configuration and increase the port block resources.
7. Verify whether a large number of broadcast or multicast packets are received on the device.
Execute the display counters rate inbound interface command to verify whether the received broadcast or multicast packet count is increasing significantly.
If the count has a large increase, use QoS to rate limit the packets and identify the reason that causes the large increase.
8. Verify whether bursts of traffic occur.
If a lot of packets are denied and dropped by the security policy, the CPU usage might be high. You can execute the following command to view whether a large number of packets are dropped.
[H3C-probe]display system internal aspf statistics zone-pair ipv4 chassis X slot X cpu 1
[H3C-probe]display system internal ip packet-drop statistics chassis X slot X cpu 1
If a large number of packets are dropped, perform the following steps:
a. Execute the following commands to determine the packet characteristics.
debug ip packet
debug ip info
debug aspf packet
b. Take actions such as permitting the packets in the security policy, configuring attack prevention policies, and QoS based on the packet characteristics
High memory usage
Symptom
The memory usage of a card remains above 70%.
Use the display memory command to display the memory usage of a card. In the command output, Total indicates total memory size, Used indicates used memory size, and FreeRatio indicate free memory ratio.
<sysname> display memory slot 2
The statistics about memory is measured in KB:
Slot 2:
Total Used Free Shared Buffers Cached FreeRatio
Mem: 16375408 2514664 13860744 0 1396 177968 84.6%
-/+ Buffers/Cache: 2335300 14040108
Swap: 0 0 0
Solution
To resolve the issue:
1. Execute the display process memory command multiple times to do the following:
¡ Display the memory usage for all processes on a card.
¡ Identify the process for which memory usage is continuously increasing.
If the memory usage of a process is continuously increasing, this process might have a memory leak. Dynamic memory is heap memory dynamically assigned to the device. Its value becomes large when memory is leaked.
<sysname> display process memory slot 2
JID Text Data Stack Dynamic Name
1 132 700 32 156 scmd
2 0 0 0 0 [kthreadd]
3 0 0 0 0 [migration/0]
4 0 0 0 0 [ksoftirqd/0]
5 0 0 0 0 [watchdog/0]
6 0 0 0 0 [migration/1]
7 0 0 0 0 [ksoftirqd/1]
8 0 0 0 0 [watchdog/1]
9 0 0 0 0 [migration/2]
10 0 0 0 0 [ksoftirqd/2]
11 0 0 0 0 [watchdog/2]
12 0 0 0 0 [migration/3]
13 0 0 0 0 [ksoftirqd/3]
14 0 0 0 0 [watchdog/3]
15 0 0 0 0 [migration/4]
16 0 0 0 0 [ksoftirqd/4]
17 0 0 0 0 [watchdog/4]
……
18919 128 76416 64 2240 diagd
……
The output shows that the process with ID 18919 uses the most memory.
2. Execute the display process memory heap command multiple times to do the following:
¡ Display heap memory usage for user process 18919.
¡ Identify the memory block for which memory usage is continuously increasing.
If the memory usage of a memory block is continuously increasing, the memory might be leaked.
<sysname> display process memory heap job 18919 verbose
Heap usage:
Size Free Used Total Free Ratio
32 541 39 580 93.3%
48 6 43 49 12.2%
64 534 32499 33033 1.6%
80 538 47 585 92.0%
112 0 534 534 0.0%
128 0 4 4 0.0%
160 0 4 4 0.0%
176 0 4 4 0.0%
256 0 2 2 0.0%
288 0 1 1 0.0%
304 0 1 1 0.0%
336 0 1 1 0.0%
688 0 4 4 0.0%
1184 0 2 2 0.0%
1456 0 2 2 0.0%
1984 0 1 1 0.0%
2032 0 2 2 0.0%
4144 0 1 1 0.0%
13792 1 0 1 100.0%
Large Memory Usage:
Used Blocks : 0
Used Memory(in bytes): 0
Free Blocks : 3
Free Memory(in bytes): 211200
Summary:
Total virtual memory heap space(in bytes) : 2490368
Total physical memory heap space(in bytes) : 2293760
Total allocated memory(in bytes) : 2170560
Related commands
|
Command |
Description |
|
display cpu-usage |
Displays current CPU usage statistics. |
|
display cpu-usage history |
Displays historical CPU usage statistics in a coordinate system. |
|
display interface |
Displays interface information. |
|
display memory |
Displays memory usage information. |
|
display process memory |
Displays memory usage information for each process on a card. |
|
display process memory heap |
Displays heap memory usage for a user process. |
|
display system internal kernel memory pool |
Displays kernel memory allocation information. |
Troubleshooting SSL VPN
Failure to log in to the SSL VPN Web interface
Symptom
The client can ping the SSL VPN gateway, but it cannot open the SSL VPN login page.
Solution
To resolve the issue:
1. Verify that a PKI domain has been specified in SSL server policy view.
[H3C] ssl server-policy XXX
[H3C-ssl-server-policy-XXX] display this
#
ssl server-policy XXX
pki-domain XXX
#
return
2. Verify that a CA certificate and local certificate have been imported to the PKI domain. Make sure the local certificate is a certificate issued by the CA to a server, not a certificate for a client.
Use the following command to view certificate information for a PKI domain:
display pki certificate domain XXXX ca
display pki certificate domain XXXX local
3. If you import certificates or modify the SSL server policy after the SSL VPN gateway is enabled, you must re-enable the SSL VPN gateway to make the configurations take effect.
Execute the following commands in sequence to re-enable the SSL VPN gateway:
¡ undo service enable
¡ service enable
4. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
ssl server-policy policy-name |
Creates an SSL server policy and enters SSL server policy view. |
|
pki-domain domain-name |
Specifies the PKI domain used by the SSL server policy. |
|
display pki certificate domain domain-name { ca | local | peer [ serial serial-num ] } |
Displays certificate information. |
|
sslvpn gateway gateway-name |
Creates an SSL VPN gateway and enters SSL VPN gateway view. |
|
service enable |
Enables the SSL VPN gateway. |
Failure to log in to the SSL VPN gateway from a browser
Symptom
A user can open the SSL VPN Web interface from a browser but cannot log in to the SSL VPN gateway.
Solution
To resolve the issue:
1. Verify that the address of the SSL VPN gateway is reachable.
¡ If ping is allowed, ping the SSL VPN gateway address from a PC.
¡ If ping is not allowed, capture packets to verify the connectivity.
2. View SSL VPN gateway information to verify the following:
¡ Verify that the SSL VPN gateway is up. If the Operation state field displays Up, the SSL VPN gateway is up. If the SSL VPN gateway is not up, enable the SSL VPN gateway from the Web interface or execute the service enable command in SSL VPN gateway view from the CLI.
¡ Verify that the SSL-related configuration is correct. By default, the device uses its self-signed certificate. To use the CA-signed certificate, apply an SSL server policy to the SSL VPN gateway. To cancel the use of the CA-signed certificate, cancel the application of the SSL server policy for the SSL VPN gateway.
¡ Verify that the SSL server policy being used by the SSL VPN gateway is the desired one.
If the used SSL server policy's configuration is edited or another SSL server policy is configured for the SSL VPN gateway, re-enable the SSL VPN gateway to make the new configuration take effect.
To re-enable the SSL VPN gateway, execute the undo service enable command and then the service enable command.
The following is an example of the SSL VPN gateway information:
[Device] display sslvpn gateway
Gateway name: gw
Operation state: Up
IP: 1.1.1.2 Port: 2000
SSL server policy configured: sslnew
SSL server policy in use: ssl
Front VPN instance: Not configured
3. View SSL VPN context information to verify the following:
¡ Verify that the SSL VPN context is up. If the Operation state field displays Up, the SSL VPN context is up. If the SSL VPN context is not up, enable the SSL VPN context from the Web interface or execute the service enable command in SSL VPN context view from the CLI.
¡ Verify that the SSL VPN context is associated with an SSL VPN gateway. The Associated SSL VPN gateway field displays the name of the SSL VPN gateway if the SSL VPN context is associated with an SSL VPN gateway. If the SSL VPN context is not associated with an SSL VPN gateway, associate one from the Web interface, or execute the gateway command in SSL VPN context view from the CLI.
The following is an example of the SSL VPN context information:
[Device] display sslvpn context
Context name: ctx
Operation state: Up
Associated SSL VPN gateway: gw
SSL client policy configured: sslnew
SSL client policy in use: ssl
4. Verify that the SSL VPN gateway listening address and ports are correctly configured on all service modules, and the listening ports on each service module is enabled.
The following is an example of TCP proxy information:
<Device> display tcp-proxy slot 1
Local Addr:port Foreign Addr:port State Service type
1.1.1.2:2000 0.0.0.0:0 LISTEN SSLVPN
5. Verify that the SSL VPN user is configured correctly.
¡ For a local user—Verify that the local user is a network access user, the service type for the user is SSL VPN, the user is authorized to access a resource group, and the resource group is well configured.
¡ For a remote user—Verify that the user's user group on the remote authentication server is configured in the SSL VPN context as a resource group. The user group and the resource group must use the same name.
6. If certificate authentication is enabled on the server and client, make sure certificates are installed correctly on the server and client.
Related commands
|
Command |
Description |
|
display tcp-proxy |
Displays brief information about TCP proxy. |
|
display sslvpn context |
Displays SSL VPN context information. |
|
display sslvpn gateway |
Displays SSL VPN gateway information. |
Failure to access internal resources from a browser
Symptom
A user cannot access internal resources after logging in to the SSL VPN gateway successfully from a browser.
Solution
To resolve the issue:
1. Verify that the access resources are configured in one of the following methods:
¡ Configure a resource list. For example:
# Create a URL item named urlitem and specify the resource URL in the URL item.
[Device-sslvpn-context-ctxweb1] url-item urlitem
[Device-sslvpn-context-ctxweb1-url-item-urlitem] url http://20.2.2.2
[Device-sslvpn-context-ctxweb1-url-item-urlitem] quit
# Create a URL list named urllist in SSL VPN context ctxweb1.
[Device-sslvpn-context-ctxweb1] url-list urllist
# Configure the heading as web for the URL list.
[Device-sslvpn-context-ctxweb1-url-list-urllist] heading web
# Assign URL item urlitem to URL list urllist.
[Device-sslvpn-context-ctxweb1-url-list-urllist] resources url-item urlitem
[Device-sslvpn-context-ctxweb1-url-list-urllist] quit
# Create an SSL VPN policy group named resourcegrp1 for SSL VPN context ctxweb1, and then add URL list urllist to the policy group for Web access.
[Device-sslvpn-context-ctxweb1] policy-group resourcegrp1
[Device-sslvpn-context-ctxweb1-policy-group-resourcegrp1] resources url-list urllist
[Device-sslvpn-context-ctxweb1-policy-group-resourcegrp1] quit
¡ Configure an ACL or a URI ACL to permit access to the internal servers, and then specify the ACL for SSL VPN Web access. For example:
[Device-sslvpn-context-ctxweb1] policy-group resourcegrp1
[Device-sslvpn-context-ctxweb1-policy-group-resourcegrp1] filter web-access acl 3000
2. Verify that the SSL VPN gateway can ping the internal resources successfully. Add routes to reach the peer devices if needed.
3. View SSL VPN gateway information to verify the following:
¡ Verify that the SSL VPN gateway is up. If the Operation state field displays Up, the SSL VPN gateway is up. If the SSL VPN gateway is not up, enable the SSL VPN gateway from the Web interface or execute the service enable command in SSL VPN gateway view from the CLI.
¡ Verify that the SSL-related configuration is correct. By default, the device uses its self-signed certificate. To use the CA-signed certificate, apply an SSL server policy to the SSL VPN gateway. To cancel the use of the CA-signed certificate, cancel the application of the SSL server policy for the SSL VPN gateway.
¡ Verify that the SSL server policy being used by the SSL VPN gateway is the desired one.
If the used SSL server policy's configuration is edited or another SSL server policy is configured for the SSL VPN gateway, re-enable the SSL VPN gateway to make the new configuration take effect.
To re-enable the SSL VPN gateway, execute the undo service enable command and then the service enable command.
The following is an example of the SSL VPN gateway information:
[Device] display sslvpn gateway
Gateway name: gw
Operation state: Up
IP: 1.1.1.2 Port: 2000
SSL server policy configured: sslnew
SSL server policy in use: ssl
Front VPN instance: Not configured
4. View SSL VPN context information to verify the following:
¡ Verify that the SSL VPN context is up. If the Operation state field displays Up, the SSL VPN context is up. If the SSL VPN context is not up, enable the SSL VPN context from the Web interface or execute the service enable command in SSL VPN context view from the CLI.
¡ Verify that the SSL VPN context is associated with an SSL VPN gateway. The Associated SSL VPN gateway field displays the name of the SSL VPN gateway if the SSL VPN context is associated with an SSL VPN gateway. If the SSL VPN context is not associated with an SSL VPN gateway, associate one from the Web interface, or execute the gateway command in SSL VPN context view from the CLI.
The following is an example of the SSL VPN context information:
[Device] display sslvpn context
Context name: ctx
Operation state: Up
Associated SSL VPN gateway: gw
SSL client policy configured: sslnew
SSL client policy in use: ssl
5. Verify that both the uplinks and downlinks operate correctly. An uplink or downlink error might occur in one of the following situations:
¡ The routes to the internal resources are not configured on the SSL VPN gateway. You can check the routing table on the device for route configuration.
¡ The internal servers do not have routes to reach the SSL VPN gateway.
¡ An address conflict occurs.
¡ An improper policy-based routing (PBR) is configured.
¡ Improper SSL VPN load balancing is configured.
¡ The device operates in dual-active mode.
To resolve this issue, change the dual-active mode to the active/standby mode, and the uplink and downlink interfaces to redundant interfaces.
Related commands
|
Command |
Description |
|
url-item |
Creates a URL item and enters its view, or enters the view of an existing URL item. |
|
url-list |
Creates a URL list and enters its view, or enters the view of an existing URL list. |
|
url |
Specifies a URL in a URL item. |
|
heading |
Configures a heading for a URL list. |
|
resources url-item |
Assigns a URL item to a URL list. |
|
policy-group |
Creates an SSL VPN policy group and enters its view, or enters the view of an existing SSL VPN policy group. |
|
resources url-list |
Assigns a URL list to an SSL VPN policy group. |
|
filter web-access acl |
Specifies an advanced ACL for Web access filtering. |
|
display sslvpn context |
Displays SSL VPN context information. |
|
display sslvpn gateway |
Displays SSL VPN gateway information. |
Failure to obtain SSL VPN gateway information from an iNode client
Symptom
A user cannot access the SSL VPN Web interface by entering the address of the SSL VPN gateway in a browser. Or, an iNode client fails to obtain the SSL VPN gateway information after the gateway address is entered.
Solution
To resolve the issue:
1. Verify that the address of the SSL VPN gateway is reachable.
¡ If ping is allowed, ping the SSL VPN gateway address from a PC.
¡ If ping is not allowed, capture packets to verify the connectivity.
2. View SSL VPN gateway information to verify the following:
¡ Verify that the SSL VPN gateway is up. If the Operation state field displays Up, the SSL VPN gateway is up. If the SSL VPN gateway is not up, enable the SSL VPN gateway from the Web interface or execute the service enable command in SSL VPN gateway view from the CLI.
¡ Verify that the SSL-related configuration is correct. By default, the device uses its self-signed certificate. To use the CA-signed certificate, apply an SSL server policy to the SSL VPN gateway. To cancel the use of the CA-signed certificate, cancel the application of the SSL server policy for the SSL VPN gateway.
¡ Verify that the SSL server policy being used by the SSL VPN gateway is the desired one.
If the used SSL server policy's configuration is edited or another SSL server policy is configured for the SSL VPN gateway, re-enable the SSL VPN gateway to make the new configuration take effect.
To re-enable the SSL VPN gateway, execute the undo service enable command and then the service enable command.
The following is an example of the SSL VPN gateway information:
[Device] display sslvpn gateway
Gateway name: gw
Operation state: Up
IP: 1.1.1.2 Port: 2000
SSL server policy configured: sslnew
SSL server policy in use: ssl
Front VPN instance: Not configured
3. View SSL VPN context information to verify the following:
¡ Verify that the SSL VPN context is up. If the Operation state field displays Up, the SSL VPN context is up. If the SSL VPN context is not up, enable the SSL VPN context from the Web interface or execute the service enable command in SSL VPN context view from the CLI.
¡ Verify that the SSL VPN context is associated with an SSL VPN gateway. The Associated SSL VPN gateway field displays the name of the SSL VPN gateway if the SSL VPN context is associated with an SSL VPN gateway. If the SSL VPN context is not associated with an SSL VPN gateway, associate one from the Web interface, or execute the gateway command in SSL VPN context view from the CLI.
The following is an example of the SSL VPN context information:
[Device] display sslvpn context
Context name: ctx
Operation state: Up
Associated SSL VPN gateway: gw
SSL client policy configured: sslnew
SSL client policy in use: ssl
4. Verify that the SSL VPN gateway listening address and ports are correctly configured on all service modules, and the listening ports on each service module is enabled.
The following is an example of TCP proxy information:
<Device> dis tcp-proxy slot 1
Local Addr:port Foreign Addr:port State Service type
1.1.1.2:2000 0.0.0.0:0 LISTEN SSLVPN
Related commands
|
Command |
Description |
|
display tcp-proxy |
Displays brief information about TCP proxy. |
|
display sslvpn context |
Displays SSL VPN context information. |
|
display sslvpn gateway |
Displays SSL VPN gateway information. |
Failure to log in to the SSL VPN gateway from an iNode client
Symptom
An iNode client obtains the SSL VPN gateway information successfully, but SSL VPN login fails.
Solution
To resolve the issue:
1. Verify that the address of the SSL VPN gateway is reachable.
¡ If ping is allowed, ping the SSL VPN gateway address from a PC.
¡ If ping is not allowed, capture packets to verify the connectivity.
2. View SSL VPN gateway information to verify the following:
¡ Verify that the SSL VPN gateway is up. If the Operation state field displays Up, the SSL VPN gateway is up. If the SSL VPN gateway is not up, enable the SSL VPN gateway from the Web interface or execute the service enable command in SSL VPN gateway view from the CLI.
¡ Verify that the SSL-related configuration is correct. By default, the device uses its self-signed certificate. To use the CA-signed certificate, apply an SSL server policy to the SSL VPN gateway. To cancel the use of the CA-signed certificate, cancel the application of the SSL server policy for the SSL VPN gateway.
¡ Verify that the SSL server policy being used by the SSL VPN gateway is the desired one.
If the used SSL server policy's configuration is edited or another SSL server policy is configured for the SSL VPN gateway, re-enable the SSL VPN gateway to make the new configuration take effect.
To re-enable the SSL VPN gateway, execute the undo service enable command and then the service enable command.
The following is an example of the SSL VPN gateway information:
[Device] display sslvpn gateway
Gateway name: gw
Operation state: Up
IP: 1.1.1.2 Port: 2000
SSL server policy configured: sslnew
SSL server policy in use: ssl
Front VPN instance: Not configured
3. View SSL VPN context information to verify the following:
¡ Verify that the SSL VPN context is up. If the Operation state field displays Up, the SSL VPN context is up. If the SSL VPN context is not up, enable the SSL VPN context from the Web interface or execute the service enable command in SSL VPN context view from the CLI.
¡ Verify that the SSL VPN context is associated with an SSL VPN gateway. The Associated SSL VPN gateway field displays the name of the SSL VPN gateway if the SSL VPN context is associated with an SSL VPN gateway. If the SSL VPN context is not associated with an SSL VPN gateway, associate one from the Web interface, or execute the gateway command in SSL VPN context view from the CLI.
The following is an example of the SSL VPN context information:
[Device] display sslvpn context
Context name: ctx
Operation state: Up
Associated SSL VPN gateway: gw
SSL client policy configured: sslnew
SSL client policy in use: ssl
4. Verify that the SSL VPN gateway listening address and ports are correctly configured on all service modules, and the listening ports on each service module is enabled.
The following is an example of TCP proxy information:
<Device> display tcp-proxy slot 1
Local Addr:port Foreign Addr:port State Service type
1.1.1.2:2000 0.0.0.0:0 LISTEN SSLVPN
5. Verify that an SSL VPN AC interface is created, an IP address is configured for the interface, and the interface is specified in the SSL VPN context for the user.
The following is an example of SSL VPN AC interface configuration:
[Device] interface SSLVPN-AC 1
[Device-SSLVPN-AC1] ip address 1.1.1.1 24
[Device-SSLVPN-AC1] quit
[Device] sslvpn context ctx
[Device-sslvpn-context-ctx] ip-tunnel interface SSLVPN-AC 1
[Device-sslvpn-context-ctx] quit
[Device] display interface SSLVPN-AC 1 brief
Brief information on interfaces in route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Primary IP Description
SSLVPN-AC1 UP UP 1.1.1.1
6. Verify that an address pool is configured, and is specified for the SSL VPN context or the authorization resource group for the user. The address pool cannot contain the address of the SSL VPN gateway.
The following is an example of address pool configuration and application:
[Device] sslvpn ip address-pool name 1.1.1.1 1.1.1.10
[Device] sslvpn context ctx
[Device-sslvpn-context-ctx] ip-tunnel address-pool name mask 24
7. Verify that the SSL VPN user is configured correctly.
¡ For a local user—Verify that the local user is a network access user, the service type for the user is SSL VPN, the user is authorized to access a resource group, and the resource group is well configured.
¡ For a remote user—Verify that the user's user group on the remote authentication server is configured in the SSL VPN context as a resource group. The user group and the resource group must use the same name.
8. If certificate authentication is enabled on the server and client, make sure certificates are installed correctly on the server and client.
9. Verify that the iNode client is of the latest version.
Related commands
|
Command |
Description |
|
display tcp-proxy |
Displays brief information about TCP proxy. |
|
display sslvpn context |
Displays SSL VPN context information. |
|
display sslvpn gateway |
Displays SSL VPN gateway information. |
|
sslvpn ip address-pool |
Creates an IPv4 address pool. |
|
ip-tunnel address-pool |
Specifies an IPv4 address pool for IP access. |
Failure to access internal resources from an iNode client
Symptom
A user cannot access internal resources after logging in to the SSL VPN gateway successfully from an iNode client.
Solution
To resolve the issue:
1. Verify that an SSL VPN AC interface is added to a security zone and is permitted by security policies.
2. Verify that the IP address of the VNIC assigned to the iNode client is added to a security zone and is permitted by security policies.
3. Configure an ACL or a URI ACL to permit access to the internal servers, and then specify the ACL for SSL VPN Web access. For example:
[Device-sslvpn-context-ctxip1] policy-group resourcegrp1
[Device-sslvpn-context-ctxip1-policy-group-resourcegrp1] filter web-access acl 3000
4. Verify that the SSL VPN gateway can ping the internal resources successfully. Add routes to reach the peer devices if needed.
5. Verify that the iNode client is of the latest version.
6. Verify that both the uplinks and downlinks operate correctly. An uplink or downlink error might occur in one of the following situations:
¡ The routes to the internal resources are not configured on the SSL VPN gateway. You can check the routing table on the device for route configuration.
¡ The internal servers do not have routes to reach the SSL VPN gateway.
¡ The device operates in dual-active mode.
¡ To resolve this issue, change the dual-active mode to the active/standby mode, and the uplink and downlink interfaces to redundant interfaces.
¡ An address conflict occurs.
¡ An improper policy-based routing (PBR) is configured.
¡ Improper SSL VPN load balancing is configured.
Related commands
|
Command |
Description |
|
policy-group |
Creates an SSL VPN policy group and enters its view, or enters the view of an existing SSL VPN policy group. |
|
filter web-access acl |
Specifies an advanced ACL for Web access filtering. |
Failure to terminate idle SSL VPN sessions of iNode client users
Symptom
The SSL VPN sessions of iNode client users do not age out even if they are idle for a long time, consuming license resources.
Solution
An iNode client periodically sends keepalive messages so the SSL VPN sessions of the iNode client users do not age out. You can configure the idle-cut feature to force users who do not access internal resources to go offline.
The idle-cut feature sets the idle-cut traffic threshold for SSL VPN sessions. SSL VPN sessions that generate traffic less than the specified threshold within the idle timeout time are terminated. The following example sets the idle-cut traffic threshold to 1000 Kilobytes:
<Device> system-view
[Device] sslvpn context ctx1
[Device-sslvpn-context-ctx1] idle-cut traffic-threshold 1000
Related commands
|
Command |
Description |
|
sslvpn context |
Creates an SSL VPN context and enters its view, or enters the view of an existing SSL VPN context. |
|
idle-cut traffic-threshold |
Sets the SSL VPN session idle-cut traffic threshold. |
User filtering, monitoring, and IP binding settings not take effect
Symptom
The ACL filtering, monitoring, and IP binding configuration in local user view for an SSL VPN user does not take effect.
Solution
To resolve the issue, configure these settings in SSL VPN context view instead of in local user view. This is because that some SSL VPN user management settings can only be configured in SSL VPN context view.
Related commands
|
Command |
Description |
|
sslvpn context |
Creates an SSL VPN context and enters its view, or enters the view of an existing SSL VPN context. |
Failure to relog in to the SSL VPN gateway
Symptom
A user fails to relog in to the SSL VPN gateway after previous successful logins.
Solution
To resolve the issue:
1. Check whether a limit is set to the number of concurrent logins for each account. In this example, the maximum number is set to 1.
[Device] sslvpn context ctx
[Device-sslvpn-context-ctx] max-onlines 1
2. You can remove the configuration of the max-onlines command if no such limit is needed. If the limit is set and you do not want to remove it, you can enable the force logout feature. When a login is attempted but logins using the account reach the maximum, this feature logs out the user with the longest idle time to allow the new login.
To configure the force logout feature, execute the following commands:
[Device] sslvpn context ctx
[Device-sslvpn-context-ctx] force-logout max-onlines enable
Related commands
|
Command |
Description |
|
sslvpn context |
Creates an SSL VPN context and enters its view, or enters the view of an existing SSL VPN context. |
|
force-logout max-onlines enable |
Enables the force logout feature. |
Failure to configure WeChat Work (or WeCom) authentication
Symptom
A user failed to access resources from the WeChat Work (or WeCom) client after WeChat Work authentication is configured.
Solution
1. Verify that the device is configured with a DNS server.
2. Verify that a trusted SSL certificate is installed.
3. Verify that the SSL VPN context enabled with WeChat Work authentication is associated to an SSL VPN gateway exclusively. For example:
[H3C]sslvpn context ctx
#
[H3C-sslvpn-context-ctx]display this
sslvpn context ctx
gateway gw domain sslvpn
4. Verify that the parameters in the SSL VPN context are configured correctly, including the API server address, company ID, app secret key, authorization policy group field name, and policy group name. If the authorization policy group is specified, the group name must be the same as the ID of the organization to which users belong on the WeChat Work management platform. If the authorization policy group is not specified, a default policy group must be configured.
[H3C]sslvpn context ctx
[H3C-sslvpn-context-ctx]display this
#
sslvpn context ctx
gateway gw domain sslvpn
wechat-work-authentication enable
wechat-work-authentication url https://qyapi.weixin.qq.com
wechat-work-authentication corp-id ww918e2ea10664acd3
wechat-work-authentication app-secret agZO0L15DmOBw-BBx9s5UmOForvCx-WEtKQWqfBQy
Ts
wechat-work-authentication authorize-field department
wechat-work-authentication open-platform-url user-defined https://open.weixin.qq.com
5. Log in to the WeChat Work management platform and verify that the redirect link to the SSL VPN gateway on the WeChat open platform is configured correctly.
Related commands
|
Command |
Description |
|
sslvpn context |
Creates an SSL VPN context and enters its view, or enters the view of an existing SSL VPN context. |
|
gateway |
Associates an SSL VPN context with an SSL VPN gateway. |
|
wechat-work-authentication enable |
Enables WeChat Work authentication. |
|
wechat-work-authentication url |
Specifies the URL of the WeChat Work API server. |
|
wechat-work-authentication corp-id |
Specifies the company ID for WeChat Work authentication. |
|
wechat-work-authentication app-secret |
Specifies the app secret key for WeChat Work authentication. |
|
wechat-work-authentication authorize-field |
Specifies the name of the authorization policy group field. |
|
wechat-work-authentication open-platform-url |
Specifies the WeChat open platform URL. |
Troubleshooting DPI
IPS or anti-virus mistakenly intercepts legal traffic
Symptom
IPS or anti-virus is configured on the firewall to protect users on the LAN against attacks. A user in the LAN fails to access the Internet and the firewall reports an IPS attack log or anti-virus attack log.
Figure 31 Network diagram
Settings on the firewall:
#
app-profile 0_IPv4
ips apply policy default mode protect
anti-virus apply policy default mode protect
#
security-policy ip
rule 0 name ips
action pass
profile 0_IPv4
#
Solution
To resolve the issue:
1. Verify that the source IP address in the IPS attack log or anti-virus attack log is the IP address of the user and the destination IP address is the IP address of the server. If yes, record the attack ID in the IPS attack log or anti-virus attack log.
2. If IPS mistakenly intercepts legal traffic, perform the following actions:
a. Create an IPS policy.
b. Disable the IPS signature that the traffic matches, or set the actions to permit and logging for the IPS signature.
c. Specify the IPS policy in the security policy applied to the firewall.
3. If anti-virus mistakenly intercepts legal traffic, perform the following actions:
a. Create an anti-virus policy.
b. Disable the virus signature that the traffic matches, or set the actions to permit and logging for the virus signature.
c. Specify the anti-virus policy in the security policy applied to the firewall.
4. Capture packets that the host sends to the server to identify whether the IPS or anti-virus attack log was mistakenly generated.
¡ If yes, modify the IPS or virus signature.
¡ If not, configure the firewall to permit traffic from the user that matches the IPS or virus signature.
Related commands
|
Command |
Description |
|
ips policy policy-name |
Creates an IPS policy and enters its view, or enters the view of an existing IPS policy. By default, an IPS policy named default exists. You cannot edit or delete the default IPS policy. |
|
signature override { pre-defined | user-defined } signature-id { { disable | enable } [ { block-source | drop | permit | redirect | reset } | capture | logging ] * } |
Changes the status and actions for an IPS signature in an IPS policy. By default, predefined IPS signatures use the actions and states defined by the system. User-defined IPS signatures use the actions and states defined in the IPS signature file from which the signatures are imported. The signature actions and status in the default IPS policy cannot be modified. |
|
anti-virus policy policy-name |
Creates an anti-virus policy and enters its view, or enters the view of an existing anti-virus policy. By default, an anti-virus policy named default exists. You cannot edit or delete the default anti-virus policy. |
|
exception signature signature-id |
Sets a signature as a signature exception. |
IPS or WAF fails to intercept attack traffic or generate attack logs
Symptom
IPS or WAF is configured on the firewall to protect users on the LAN against attacks. However, an attacker can successfully launch an attack (such as cross-site script attack or brute force attack) from the Internet to the internal target server and crack the password of the server. The firewall does not generate an attack log.
Figure 32 Network diagram
Settings on the firewall:
#
app-profile 0_IPv4
ips apply policy default mode protect
waf apply policy default mode protect
#
security-policy ip
rule 0 name ips
action pass
profile 0_IPv4
#
Solution
To resolve the issue:
1. Verify that the firewall has installed with the license for IPS or WAF.
2. Verify that DPI is operating correctly on the device.
[H3C] display inspect status
Chassis 0 Slot 1:
Running status: normal
3. Verify that the firewall uses the most up-to-date IPS signature library or most up-to-date WAF signature library.
<sysname> display ips signature library
IPS signature library information:
Type SigVersion ReleaseTime Size
Current 1.0.81 Thu Oct 31 08:35:05 2019 4639264
Last 1.0.80 Sat Oct 12 07:58:23 2019 4565664
Factory 1.0.0 Fri Dec 28 06:27:33 2018 76496
<sysname>display waf signature library
WAF signature library information:
Type SigVersion ReleaseTime Size(bytes)
Current 1.0.2 Thu Oct 31 03:22:10 2019 1018752
Last 1.0.0 Fri Dec 28 08:53:30 2018 19824
Factory 1.0.0 Fri Dec 28 08:53:30 2018 19824
4. Verify that IPS policy settings or WAF policy settings have been activated. You can activate the IPS policy settings or WAF policy settings from the CLI (by using the inspect activate command) or from the Web interface.
[H3C-probe] display system internal inspect dim-rule
Slot 1:
MdcID MoudleName Total MD5 rules
0 Anti-Virus 0
MdcID RuleID ModuleName L4ProName uiAppIdL5
0 1 IPS TCP HTTP
0 2147483649 FFILTER TCP
0 2 IPS TCP HTTP
0 2147483650 FFILTER TCP
0 2147483651 FFILTER TCP
0 4 IPS TCP HTTP
0 2147483652 FFILTER TCP
0 5 IPS TCP HTTP
[H3C-probe]display system internal inspect dim-rule | include WAF
0 1 WAF TCP HTTP
0 16 WAF TCP HTTP
0 37 WAF TCP HTTP
0 38 WAF TCP HTTP
0 43 WAF TCP HTTP
5. Verify that a session that meets the following requirements has been established between the host and the server:
¡ The source IP address and destination IP address are in the specified security zone.
¡ DPI with the IPS policy or WAF policy specified is enabled in the security zone.
# Display information about the IPv4 unicast session initiated by the host.
[H3C] display session table ipv4 source-ip 1.1.1.101 verbose
Slot 1:
Initiator:
Source IP/port: 1.1.1.101/34679
Destination IP/port: 2.2.2.12/5190
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet1/0/10
Source security zone: Trust
Responder:
Source IP/port: 2.2.2.12/5190
Destination IP/port: 1.1.1.101/34679
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet1/0/11
Source security zone: Untrust
State: TCP_ESTABLISHED
Application: AOL
Start time: 2016-01-21 16:13:16 TTL: 1194s
Initiator->Responder: 3 packets 930 bytes
Responder->Initiator: 1 packets 92 bytes
Total sessions found: 1
# Display the match statistics for DPI engine inspection rules.
[H3C-probe]display system internal inspect hit-statistics
Slot 1:
Rule ID Module Rule hits AC hits PCRE try PCRE hits
5041 APR 0 3 0 0
5126 APR 0 9 0 0
5127 APR 0 9 0 0
8584 IPS 1 2 0 0
9410 APR 0 1 0 0
21768 IPS 0 2 0 0
21852 IPS 1 2 0 0
22114 IPS 0 2 0 0
22406 IPS 1 1 0 0
23089 IPS 2 2 4 2
23213 IPS 0 4 2 2
23271 IPS 0 2 1 0
23341 IPS 1 2 1 1
23722 IPS 2 8 2 2
23804 IPS 0 1 0 0
18096 WAF 0 4 2 0
23311 WAF 1 14 1 1
23791 WAF 0 2 1 0
23915 WAF 0 8 4 0
6. Verify that the IPS signature or WAF signature that the traffic hits has been enabled and actions have been specified for the signature.
¡ You can use the signature override pre-defined 8 enable reset logging command to enable the IPS signature and specify actions for the signature.
¡ You can use the signature override pre-defined pre-defined 56 enable reset logging command to enable a predefined WAF signature and specify actions for the signature.
[H3C]display ips signature pre-defined 8
Type : Pre-defined
Signature ID: 8
Status : Disable
Action : Permit & Logging
Name : (MS11-015)DVR-MS_Vulnerability
Protocol : TCP
Severity : Critical
Fidelity : Medium
Direction : To-client
Category : Vulnerability
Reference : CVE-2011-0042;MS11-015;
[H3C]display waf signature pre-defined 56
Type : Pre-defined
Signature ID: 56
Status : Disable
Action : Permit & Logging
Name : CVE-2012-3351_LongTail_JW_Player_XSS_Vulnerability
Protocol : TCP
Severity : Medium
Fidelity : Medium
Direction : To-server
Category : Vulnerability
Reference : CVE-2012-3351;
7. If the issue persists, it might indicate that the IPS signature library or WAF signature library does not support this attack. In this case, capture the attack packets and contact H3C Support.
Related commands
|
Command |
Description |
|
ips policy policy-name |
Creates an IPS policy and enters its view, or enters the view of an existing IPS policy. By default, an IPS policy named default exists. You cannot edit or delete the default IPS policy. |
|
waf policy policy-name |
Creates a WAF policy and enters its view, or enter the view of an existing WAF policy. By default, a WAF policy named default exists. You cannot edit or delete the default WAF policy. |
|
signature override { pre-defined | user-defined } signature-id { { disable | enable } [ { block-source | drop | permit | redirect | reset } | capture | logging ] * } |
Changes the status and actions for an IPS or WAF signature in an IPS or WAF policy. By default, predefined IPS or WAF signatures use the actions and states defined by the system. User-defined IPS or WAF signatures use the actions and states defined in the IPS or WAF signature file from which the signatures are imported. The signature actions and status in the default IPS or WAF policy cannot be modified. |
|
inspect activate |
Activates the policy and rule configurations for DPI service modules. By default, the creation, modification, and deletion of DPI service policies and rules do not take effect. |
|
display system internal inspect hit-statistics [ module-id ] [ rule-id ] [ slot slot-number [ cpu cpu-number ] ] |
Displays the match statistics for DPI engine inspection rules. |
|
display inspect status |
Displays the status of the DPI engine. |
Application rate limit does not take effect
Symptom
Bandwidth management is configured on the firewall to limit the downstream traffic rate of Thunder, but the downstream traffic rate of Thunder is not correctly limited.
Figure 33 Network diagram
Settings on the firewall:
traffic-policy
rule 1 name Thunder
action qos profile Thunder_20M
source-zone Trust
destination-zone Untrust
application app-group 1
profile name thunder_20m
bandwidth downstream maximum 20000
bandwidth upstream maximum
Solution
To resolve the issue:
1. Verify that the firewall uses the most up-to-date APR signature library. To install the most up-to-date APR signature library, access the company's website and download the signature file.
2. Verify that the DPI engine is enabled. You can use the undo inspect bypass command to enable the DPI engine.
3. Verify that the application audit and management policy settings have been activated. You can use the inspect activate command to activate application audit and management policy settings.
[H3C-probe] display system internal inspect dim-rule
Slot 1:
MdcID MoudleName Total MD5 rules
0 Anti-Virus 0
MdcID RuleID ModuleName L4ProName uiAppIdL5
1 1 AUDIT TCP WECHAT_LOGIN_IOS
_TCP_M
0 1 IPS TCP HTTP
0 2147483649 FFILTER TCP
1 2 AUDIT TCP WECHAT_LOGIN_AND
ROID_TCP_M
0 2 IPS TCP HTTP
0 2147483650 FFILTER TCP
1 3 AUDIT TCP WECHAT_SENDTEXT_
WINDOWS_TCP_M
0 2147483651 FFILTER TCP
1 4 AUDIT TCP WECHAT_SENDTEXT_
IOS_TCP_M
0 4 IPS TCP HTTP
4. Verify that the application audit and management policy has been enabled and the traffic can be processed based on the policy rule. You can use the rule move Thunder before b command to move the traffic rule to a new position to ensure that the firewall uses the policy rule to process the downstream traffic of Thunder.
5. Verify that application signatures in the session are added to user-defined application groups and the application group has been applied with the traffic policy.
<sysname> display session table ipv4 verbose
Slot 1:
Initiator:
Source IP/port: 1.1.1.195/51353
Destination IP/port: 2.2.2.51/59287
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet1/0/10
Source security zone: Trust
Responder:
Source IP/port: 2.2.2.51/59287
Destination IP/port: 1.1.1.195/51353
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet1/0/11
Source security zone: Untrust
State: TCP_SYN_RECV
Application: GENERAL_TCP
Start time: 2016-01-21 17:51:44 TTL: 951s
Initiator->Responder: 1 packets 56 bytes
Responder->Initiator: 1 packets 56 bytes
If most applications in the session are GENERAL_TCP or GENERAL_UDP, it indicates that new signatures are required to be defined for Thunder. In this case, contact H3C Support.
Related commands
|
Command |
Description |
|
traffic-policy |
Enters traffic policy view. |
|
rule move rule-name1 { after | before } rule-name2 |
Moves a traffic rule to a new position. |
|
display traffic-policy statistics bandwidth { downstream | total | upstream } { per-ip { ipv4 [ ipv4-address ] | ipv6 [ ipv6-address ] } rule rule-name | per-rule [ name rule-name ] | per-user [ user user-name ] rule rule-name } |
Displays traffic statistics for traffic rules. |
File filtering or data filtering does not take effect
Symptom
File filtering is configured on the firewall to block and log confidential files transferred between the LAN and the Internet. However, a user in the LAN still can upload a confidential file to the Internet, and the firewall does not generate a file filtering log.
Data filtering is configured on the firewall to ensure security of the data transferred between the LAN and the Internet. However, a user in the LAN still can upload sensitive data such as bank card numbers and ID card numbers to the Internet. No data filtering log is generated.
Figure 34 Network diagram
Settings on the firewall:
#
file-filter policy ffilter
rule ffilter
filetype-group ffilter
application all
direction both
action drop logging
#
file-filter filetype-group ffilter
pattern 0 text pe
pattern 1 text elf
pattern 10 text vsdx
pattern 11 text msg
pattern 12 text pub
pattern 13 text zip
pattern 14 text rar
pattern 15 text tar.gz
pattern 16 text tgz
pattern 2 text doc
pattern 3 text pdf
pattern 4 text xls
pattern 5 text ppt
pattern 6 text docx
pattern 7 text xlsx
pattern 8 text pptx
pattern 9 text vsd
#
data-filter keyword-group dfilter
pre-defined-pattern name bank-card-number
pre-defined-pattern name credit-card-number
pre-defined-pattern name id-card-number
pre-defined-pattern name phone-number
#
data-filter policy dfilter
rule dfilter
keyword-group dfilter
application all
direction both
action drop logging
#
app-profile 0_IPv4
file-filter apply policy ffilter
data-filter apply policy dfilter
#
security-policy ip
rule 0 name ffilter
action pass
profile 0_IPv4
#
Solution
To resolve the issue:
1. Verify that DPI is operating correctly on the device.
[H3C] display inspect status
Chassis 0 Slot 1:
Running status: normal
The normal running status indicates that DPI is correctly operating.
2. Verify that the file type group includes the type of the transferred file.
3. Verify that the used application layer protocols have been specified for file filtering rules in the file filtering policy or for data filtering rules in the data filtering policy. File filtering rules or data filtering rules can be applied to the following protocols: HTTP, FTP, SMTP, IMAP, NFS, POP3, RTMP, and SMB.
4. Verify that the file filtering policy settings or data filtering policy settings have been activated.
A file filtering rule with a 10-digit ID is a predefined file filtering rule. You can activate predefined file filtering policy settings from the CLI (by using the inspect activate command in system view) or from the Web interface.
You can activate data filtering policy settings from the CLI (by using the inspect activate command in system view) or from the Web interface.
[H3C-probe] display system internal inspect dim-rule | include FFILTER
23 FFILTER TCP HTTP
0 2147483671 FFILTER TCP
1 24 FFILTER TCP FTP
0 2147483672 FFILTER TCP
1 25 FFILTER TCP SMTP
0 2147483673 FFILTER TCP
1 26 FFILTER TCP IMAP
0 2147483674 FFILTER TCP
1 27 FFILTER TCP POP3
0 2147483675 FFILTER TCP
1 28 FFILTER TCP NFS
0 2147483676 FFILTER TCP
1 29 FFILTER TCP MICROSOFT-DS
1 30 FFILTER TCP RTMP
[H3C-probe]display system internal inspect dim-rule | include DFILTER
1 24 DFILTER TCP HTTP
1 25 DFILTER TCP FTP-DATA
1 26 DFILTER TCP SMTP
1 27 DFILTER TCP IMAP
1 28 DFILTER TCP POP3
1 29 DFILTER TCP NFS
1 30 DFILTER TCP MICROSOFT-DS
1 31 DFILTER TCP RTMP
5. Verify that a session that meets the following requirements has been established between the host and the server:
¡ The source IP address and destination IP address are in the specified security zone.
¡ DPI with the file filtering policy or data filtering policy specified is enabled in the security zone.
# Display information about the IPv4 unicast session initiated by the host.
[H3C-probe] display session table ipv4 source-ip 7.0.1.2 verbose
Slot 2:
Initiator:
Source IP/port: 7.0.1.2/50779
Destination IP/port: 7.0.0.2/80
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 7.0.0.2/80
Destination IP/port: 7.0.1.2/50779
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/3
Source security zone: Untrust
State: TCP_ESTABLISHED
Application: HTTP
Rule ID: 0
Rule name: ips
Start time: 2019-11-15 11:31:01 TTL: 1197s
Initiator->Responder: 7 packets 1073 bytes
Responder->Initiator: 7 packets 2413 bytes
Total sessions found: 1
6. Display the match statistics for DPI engine inspection rules.
[H3C-probe]display system internal inspect hit-statistics
Slot 2:
Rule ID Module Rule hits AC hits PCRE try PCRE hits
2147483650 FFILTER 2 2 0 0
2147483657 FFILTER 1 1 0 0
2147483669 FFILTER 2 2 0 0
3432 APR 2 2 0 0
If only predefined file filtering rules have hit counts, the transferred files might carry false extensions. You can use the file-filter false-extension action drop command to set the drop action for such files, and then observe whether the firewall can block and log the files.
7. If the issue persists, it might indicate that the firewall does not support the encoding format of the files. In this case, capture the packets exchanged between the host and the firewall and then contact H3C Support.
Related commands
|
Command |
Description |
|
file-filter policy policy-name |
Creates a file filtering policy and enters its view, or enters the view of an existing file filtering policy. By default, a file filtering policy named default exists. You cannot edit or delete the default file filtering policy. |
|
filetype-group group-name |
Applies a file type group to a file filtering rule. By default, a file filtering rule contains a file type group named default. |
|
inspect activate |
Activates the policy and rule configurations for DPI service modules. By default, the creation, modification, and deletion of DPI service policies and rules do not take effect. |
|
display system internal inspect hit-statistics [ module-id ] [ rule-id ] [ slot slot-number [ cpu cpu-number ] ] |
Displays the match statistics for DPI engine inspection rules. |
|
display inspect status |
Displays the status of the DPI engine. |
|
file-filter false-extension action { drop | permit } |
Sets the action for packets with files carrying false extensions. |
|
data-filter apply policy policy-name |
Applies a data filtering policy to a DPI application profile. By default, no data filtering policy is applied to a DPI application profile. |
|
data-filter keyword-group keywordgroup-name |
Creates a keyword group and enters its view, or enters the view of an existing keyword group. |
SSL decryption does not take effect
Symptom
An SSL-decryption proxy policy and IPS are configured on the firewall to provide secure HTTPS transport for users on the LAN and Internet. However, an attacker can successfully launch an encrypted HTTPS attack (such as cross-site script attack and brute force attack) from the Internet and crack the password of the target server. IPS on the firewall does not intercept the attack or generate an attack log. The SSL decryption feature does not take effect.
Figure 35 Network diagram
Settings on the firewall:
#
app-proxy-policy
rule 1 name ssl-proxy
action ssl-decrypt
#
app-profile 0_IPv4
ips apply policy default mode protect
#
security-policy ip
rule 0 name ips
action pass
profile 0_IPv4
#
Solution
1. Verify that the firewall can intercept non-encrypted HTTP traffic.
If the firewall fails to intercept non-encrypted HTTP traffic, resolve the issue according to "IPS or WAF fails to intercept attack traffic or generate attack logs".
If the firewall can intercept non-encrypted HTTP traffic, proceed to the next step.
2. Verify that the firewall can implement SSL proxy.
[H3C]display app-proxy server-certificate
Slot 1:
Total server certificates: 1
Certificate info: BreakingPoint_serverA_2048.server.int
Proxy count: 6996
Most recent proxy time: 2019/11/18 10:23:48
First proxy at: 2019/11/15 17:21:12
3. Verify that the network is a Layer 3 network. In the current software version, SSL decryption is not supported on the Layer 2 network.
4. Verify that DPI is operating correctly on the device.
[H3C] display inspect status
Chassis 0 Slot 1:
Running status: normal
The normal running status indicates that DPI is correctly operating.
5. Verify that the HTTPS server is not in the user-defined SSL hostname whitelist.
[H3C] display app-proxy ssl whitelist hostname predefined
Chrome HSTS-defined hostnames:
status Hostname
enabled 2mdn.net
enabled accounts.firefox.com
enabled aclu.org
enabled activiti.alfresco.com
enabled adamkostecki.de
enabled addvocate.com
enabled adsfund.org
enabled aie.de
...
<sysname> display app-proxy ssl whitelist ip all
Slot 1:
IP address Port
--------------------------
9.9.9.5 443
9.9.9.6 443
9.9.9.7 443
9.9.9.8 443
9.9.9.9 443
9.9.9.10 443
9.9.9.11 443
9.9.9.12 443
If the HTTP server is in the whitelist, remove the hostname of the server from the user-defined SSL hostname whitelist by using the following commands:
[H3C] undo app-proxy ssl whitelist user-defined-hostname
<sysname> reset app-proxy ssl whitelist ip
[H3C] app-proxy ssl whitelist activate
6. Verify that the traffic is not forwarded across cards. SSL decryption does not support decrypting inter-card traffic.
<sysname>display session table ipv4 source-ip 7.0.1.2 verbose
Slot 1:
Initiator:
Source IP/port: 7.0.1.2/55933
Destination IP/port: 8.8.8.2/443
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 8.8.8.2/443
Destination IP/port: 7.0.1.2/55933
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Reth1
Source security zone: Trust
State: INACTIVE
Application: HTTPS
Rule ID: 0
Rule name: ips
Start time: 2019-11-18 10:59:43 TTL: 299s
Initiator->Responder: 0 packets 0 bytes
Responder->Initiator: 0 packets 0 bytes
Initiator:
Source IP/port: 7.0.1.2/55852
Destination IP/port: 8.8.8.2/80
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 8.8.8.2/80
Destination IP/port: 7.0.1.2/55852
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Reth1
Source security zone: Trust
State: INACTIVE
Application: HTTP
Rule ID: 0
Rule name: ips
Start time: 2019-11-18 10:59:02 TTL: 257s
Initiator->Responder: 0 packets 0 bytes
Responder->Initiator: 0 packets 0 bytes
Initiator:
Source IP/port: 7.0.1.2/55932
Destination IP/port: 8.8.8.2/443
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 8.8.8.2/443
Destination IP/port: 7.0.1.2/55932
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Reth1
Source security zone: Trust
State: INACTIVE
Application: HTTPS
Rule ID: 0
Rule name: ips
Start time: 2019-11-18 10:59:43 TTL: 299s
Initiator->Responder: 0 packets 0 bytes
Responder->Initiator: 0 packets 0 bytes
Total sessions found: 3
Slot 2:
Initiator:
Source IP/port: 7.0.1.2/55933
Destination IP/port: 8.8.8.2/443
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 8.8.8.2/443
Destination IP/port: 7.0.1.2/55933
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Reth1
Source security zone: Trust
State: TCP_TIME_WAIT
Application: HTTPS
Rule ID: 0
Rule name: ips
Start time: 2019-11-18 10:59:43 TTL: 0s
Initiator->Responder: 6 packets 776 bytes
Responder->Initiator: 7 packets 899 bytes
Initiator:
Source IP/port: 7.0.1.2/55852
Destination IP/port: 8.8.8.2/80
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 8.8.8.2/80
Destination IP/port: 7.0.1.2/55852
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Reth1
Source security zone: Trust
State: TCP_ESTABLISHED
Application: HTTP
Rule ID: 0
Rule name: ips
Start time: 2019-11-18 10:59:02 TTL: 1157s
Initiator->Responder: 8 packets 1256 bytes
Responder->Initiator: 9 packets 3456 bytes
Initiator:
Source IP/port: 7.0.1.2/55932
Destination IP/port: 8.8.8.2/443
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 8.8.8.2/443
Destination IP/port: 7.0.1.2/55932
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Reth1
Source security zone: Trust
State: TCP_TIME_WAIT
Application: HTTPS
Rule ID: 0
Rule name: ips
Start time: 2019-11-18 10:59:43 TTL: 1s
Initiator->Responder: 7 packets 816 bytes
Responder->Initiator: 7 packets 899 bytes
Total sessions found: 3
7. If the issue persists, it might indicate that the firewall does not support defending against this type of encryption attack. In this case, capture the packets exchanged between the host and the firewall and then contact H3C Support.
Related commands
|
Command |
Description |
|
app-proxy-policy |
Enters proxy policy view. |
|
app-proxy ssl whitelist user-defined-hostname host-name |
Adds a hostname to the user-defined SSL hostname whitelist. If the DNS Name or Common Name value in a server certificate contains a hostname on the SSL hostname whitelist, the device does not proxy the SSL connections destined for the server. |
|
display app-proxy ssl whitelist ip { all | ip-address } |
Displays the SSL IP address whitelist. |
|
display inspect status |
Displays the status of the DPI engine. |
Application audit and management does not take effect
Symptom
Application audit and management is configured on the firewall to ensure security of the data transferred between the LAN and the Internet. However, a user can successfully perform a sensitive behavior supposed to be denied by the audit policy (such as file transfer and login) and the firewall does not generate an audit log.
Figure 36 Network diagram
Settings on the firewall:
#
uapp-control
policy name default audit
rule 1 app-category IM behavior FileTransfer bhcontent any keyword include any
action deny audit-logging
#
Solution
To resolve the issue:
1. Verify that the device uses the most up-to-date APR signature library. To install the most up-to-date APR signature library, access the company's website and download the signature file.
2. Verify that the DPI engine is enabled. You can use the undo inspect bypass command to enable the DPI engine.
3. Verify that the application audit and management policy settings have been activated. You can active the application audit and management policy settings from the CLI (by using the inspect activate command in system view) or from the Web interface.
[H3C-probe] display system internal inspect dim-rule
Slot 1:
MdcID MoudleName Total MD5 rules
0 Anti-Virus 0
MdcID RuleID ModuleName L4ProName uiAppIdL5
1 1 AUDIT TCP WECHAT_LOGIN_IOS
_TCP_M
0 1 IPS TCP HTTP
0 2147483649 FFILTER TCP
1 2 AUDIT TCP WECHAT_LOGIN_AND
ROID_TCP_M
0 2 IPS TCP HTTP
0 2147483650 FFILTER TCP
1 3 AUDIT TCP WECHAT_SENDTEXT_
WINDOWS_TCP_M
0 2147483651 FFILTER TCP
1 4 AUDIT TCP WECHAT_SENDTEXT_
IOS_TCP_M
0 4 IPS TCP HTTP
4. Verify that the application audit and management policy has been enabled and the traffic can be processed based on the policy rules.
5. Verify that a session that meets the following requirements has been established between the host and the server:
¡ The source IP address and destination IP address are in the specified security zone.
¡ DPI with the application audit and management policy specified is enabled in the security zone.
[H3C-probe]display session table ipv4 source-ip 7.0.1.2 verbose
Slot 2:
Initiator:
Source IP/port: 7.0.1.2/50779
Destination IP/port: 7.0.0.2/80
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 7.0.0.2/80
Destination IP/port: 7.0.1.2/50779
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/3
Source security zone: Untrust
State: TCP_ESTABLISHED
Application: HTTP
Rule ID: 0
Rule name: ips
Start time: 2019-11-15 11:31:01 TTL: 1197s
Initiator->Responder: 7 packets 1073 bytes
Responder->Initiator: 7 packets 2413 bytes
Total sessions found: 1
6. If the issue persists, it might indicate that the firewall does not support auditing this application. In this case, capture the packets exchanged between the host and the firewall and then contact H3C Support.
Related commands
|
Command |
Description |
|
inspect activate |
Activates the policy and rule configurations for DPI service modules. By default, the creation, modification, and deletion of DPI service policies and rules do not take effect. |
|
display inspect status |
Displays the status of the DPI engine. |
URL filtering does not take effect
Symptom
URL filtering is configured on the firewall to provide restricted Web access from LAN users to the Internet. However, a user can successfully visit a potentially harmful website (such as a porn website) and the firewall does not generate a URL filtering log.
Figure 37 Network diagram
Settings on the firewall:
#
url-filter policy url
default-action permit logging
category Pre-Botnet action reset logging
category Pre-ChildAbuse action reset logging
category Pre-CriminalActivity action reset logging
category Pre-Discrimination action reset logging
category Pre-Divining action reset logging
category Pre-Drugs action reset logging
category Pre-Gamble action reset logging
category Pre-Hacking action reset logging
category Pre-IllegalSoftware action reset logging
category Pre-Lottery action reset logging
category Pre-MaliciousURL action reset logging
category Pre-Phishing action reset logging
category Pre-Pornography action reset logging
category Pre-Religion action reset logging
category Pre-SchoolCheating action reset logging
category Pre-Spam action reset logging
category Pre-Suicide action reset logging
category Pre-Violence action reset logging
#
app-profile 0_IPv4
url-filter apply policy url
#
security-policy ip
rule 0 name url
action pass
counting enable
profile 0_IPv4
#
Solution
To resolve the issue:
1. Verify that the device uses the most up-to-date URL filtering signature library. To install the most up-to-date URL filtering signature library, access the company's website and download the signature file.
2. Verify that the DPI engine is enabled. You can use the undo inspect bypass command to enable the DPI engine.
3. Verify that the visited webpage is an HTTPS encrypted webpage. If yes, enable SSL decryption on the firewall.
4. Verify that the URL filtering policy settings have been activated. You can active the URL filtering policy settings from the CLI (by using the inspect activate command in system view) or from the Web interface.
[H3C-probe] display system internal inspect dim-rule
Slot 1:
MdcID MoudleName Total MD5 rules
0 Anti-Virus 0
MdcID RuleID ModuleName L4ProName uiAppIdL5
0 356581376 UFLT TCP HTTP
0 268435456 UFLT TCP HTTP
0 356646912 UFLT TCP HTTP
0 268435457 UFLT TCP HTTP
0 431030273 UFLT TCP HTTP
0 384958465 UFLT TCP HTTP
0 2147483649 FFILTER TCP
0 447873026 UFLT TCP HTTP
0 268435458 UFLT TCP HTTP
5. Verify that a session that meets the following requirements has been established between the host and the server:
¡ The source IP address and destination IP address are in the specified security zone.
¡ DPI with the URL filtering policy specified is enabled in the security zone.
[H3C-probe]display session table ipv4 source-ip 7.0.1.2 verbose
Slot 2:
Initiator:
Source IP/port: 7.0.1.2/50779
Destination IP/port: 7.0.0.2/80
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/2
Source security zone: Trust
Responder:
Source IP/port: 7.0.0.2/80
Destination IP/port: 7.0.1.2/50779
DS-Lite tunnel peer: -
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: GigabitEthernet2/0/3
Source security zone: Untrust
State: TCP_ESTABLISHED
Application: HTTP
Rule ID: 0
Rule name: ips
Start time: 2019-11-15 11:31:01 TTL: 1197s
Initiator->Responder: 7 packets 1073 bytes
Responder->Initiator: 7 packets 2413 bytes
Total sessions found: 1
6. Verify that the URL that the user visits exactly matches a user-defined URL category.
7. If the issue persists, it might indicate that the website URL is not included in the URL filtering signature library. In this case, capture the packets exchanged between the host and the firewall and then contact H3C Support.
Related commands
|
Command |
Description |
|
url-filter apply policy policy-name |
Applies a URL filtering policy to a DPI application profile. By default, no URL filtering policy is applied to a DPI application profile. |
|
inspect activate |
Activates the policy and rule configurations for DPI service modules. By default, the creation, modification, and deletion of DPI service policies and rules do not take effect. |
|
display inspect status |
Displays the status of the DPI engine. |
Server connection detection does not take effect
Symptom
Server connection detection (SCD) is configured on the firewall to block and log illegal connections initiated by the protected servers. However, the firewall fails to identify and log such connections.
Figure 38 Network diagram
Settings on the firewall:
#
scd policy name default-7.0.0.2
protected-server 7.0.0.2
logging enable
policy enable
rule 1
permit-dest-ip 7.0.0.255
protocol udp port 137 to 138
#
Solution
To resolve the issue:
1. Verify that fast log output is disabled for SCD on the firewall.
Fast log output and syslog output are mutually exclusive. To output SCD logs (syslog), disable fast log output for SCD.
2. Verify that the following conditions exist:
¡ The SCD policy has been enabled and logging has been enabled for illegal connections initiated by the protected server in the policy.
¡ The source IP address of the illegal connection matches the IP address of the protected server. The destination IP addresses of the illegal connection matches the destination IP address criterion for a rule in the SCD policy.
<sysname> display scd policy
Id Name Protected server Rules Logging Policy status
1 12 1.2.2.3 1 Enabled Enabled
2 default-7.0.0.2 7.0.0.2 1 Enabled Enabled
3. Verify the traffic matching order against all the security policy rules. Make sure the firewall processes the traffic of the illegal connection based on the SCD policy rather than other policies.
4. If the issue persists, it might indicate that the firewall does not support identifying this type of traffic. In this case, capture the packets exchanged between the host and the server and then contact H3C Support.
Related commands
|
Command |
Description |
|
scd policy name policy-name |
Creates an SCD policy and enter its view, or enter the view of an existing SCD policy. |
|
display scd policy [ name policy-name ] |
Displays SCD policy information. |
IP reputation does not take effect
Symptom
IP reputation is configured on the firewall to prevent attacks initiated from IP addresses with bad reputations. However, an Internet user at an IP address on the IP reparation list can successfully communicate with a local user, and the firewall does not generate an IP-reputation log.
Figure 39 Network diagram
Settings on the firewall:
#
ip-reputation
global enable
top-hit-statistics enable
attack-category 1 action deny logging enable
attack-category 2 action deny logging disable
attack-category 3 action deny logging enable
attack-category 4 action deny logging enable
attack-category 5 action deny logging enable
attack-category 6 action deny logging enable
attack-category 7 action deny logging enable
attack-category 8 action deny logging enable
attack-category 9 action deny logging enable
attack-category 10 action deny logging enable
attack-category 11 action deny logging enable
attack-category 12 action deny logging enable
attack-category 13 action deny logging enable
attack-category 14 action deny logging enable
attack-category 15 action deny logging enable
attack-category 16 action deny logging enable
attack-category 17 action deny logging enable
attack-category 18 action deny logging enable
attack-category 19 action deny logging enable
attack-category 20 action deny logging enable
attack-category 21 action deny logging enable
attack-category 22 action deny logging enable
#
Solution
To resolve the issue:
1. Verify that the firewall has been installed with the license for IP reputation.
2. Verify that the IP address of the Internet user has not been configured as an exception IP address for IP reputation.
3. Verify that deny is specified as the action to be taken on packets matching the attack category for the IP address and logging is enabled for the attack category.
[H3C-ip-reputation] display ip-reputation attack-category
Attack id Attack name Action Logging
---------------------------------------------------
1 C&C deny enable
2 Network_Worm deny disable
3 Risk_Software deny enable
4 Malware deny enable
5 Trojan deny enable
6 Infectious_Virus deny enable
4. If the issue persists, it might indicate that the IP address of the Internet user is not included in the IP reputation library. In this case, capture the packets exchanged between the host and the server and then contact H3C Support.
Related commands
|
Command |
Description |
|
display ip-reputation attack-category |
Displays information about attack categories for IP reputation. If you do not specify actions for an attack category, the pre-defined actions are displayed. |
|
display ip-reputation exception |
Displays exception IP addresses. The command displays exception IP addresses, if any, when the IP reputation is enabled. |
Data analysis center fails to display or refresh logs
Symptom
DPI services are configured on the firewall to ensure security of the information transferred between the LAN and Internet. The data analysis center is configured to collect log data from various services for central analysis and reporting. However, the data analysis center fails to display logs generated for DPI services or fails to refresh logs.
Figure 40 Network diagram
Settings on the firewall:
#
app-profile 0_IPv4
ips apply policy default mode protect
data-filter apply policy default
url-filter apply policy default
file-filter apply policy default
anti-virus apply policy default mode protect
#
security-policy ip
rule 0 name 1
action pass
profile 0_IPv4
source-zone Trust
source-zone Untrust
destination-zone Trust
destination-zone Untrust
#
Solution
To resolve the issue:
1. Verify that DPI is operating correctly on the device.
[H3C] display inspect status
Chassis 0 Slot 1:
Running status: normal
2. Display the match statistics for DPI engine inspection rules.
[H3C-probe]display system internal inspect hit-statistics
Slot 1:
Rule ID Module Rule hits AC hits PCRE try PCRE hits
0 FFILTER 0 78225 0 0
0 DFILTER 0 545415 0 0
1 FFILTER 0 78225 0 0
1 DFILTER 0 545415 0 0
2 FFILTER 52341 78225 52341 52341
2 DFILTER 0 545415 0 0
3 FFILTER 0 78225 0 0
3 DFILTER 0 545415 0 0
4 FFILTER 25884 78225 25884 25884
4 DFILTER 0 545415 0 0
2147483652 FFILTER 359139 359139 0 0
5 FFILTER 0 78225 0 0
5 DFILTER 0 545415 0 0
2147483653 FFILTER 9 9 0 0
6 FFILTER 0 78225 0 0
6 DFILTER 0 545415 0 0
2147483654 FFILTER 207554 207554 0 0
7 FFILTER 0 78225 0 0
7 DFILTER 0 545415 0 0
2147483656 FFILTER 159715 159715 0 0
2147483657 FFILTER 985048 985048 0 0
3. Wait for a period of time to check whether the firewall generates logs.
4. Verify that the time on the firewall and the local PC are consistent. You can change the system time from the CLI (by using the clock datetime command) or from the Web interface.
<sysname> display clock
18:37:21 UTC Tue 11/26/2019
5. Verify that the firewall is enabled with session statistics collection for software fast forwarding. You can use the session statistics enable command to enable this feature. This feature must be enabled to output some types of logs such as flow log.
6. Verify that the firewall is enabled with URL filtering logging for access to resources of predefined resource types including css, gif, ico, jpg, js, png, swf, and xml. You can use the undo url-filter log except pre-defined { css | gif | ico | jpg | js | png | swf | xml } command to enable URL filtering logging for these resource types.
7. Verify that the storage time limit, storage space usage limit, and the storage limit-triggered action are set appropriately for each service. To set the settings, use the dac storage service service-type service-name limit { hold-time time-value | usage usage-value | action { delete | log-only } } command. You can restore the default settings.
8. If the issue persists, it might indicate that the ntopd process has an exception. In this case, capture the packets exchanged and then contact H3C Support.
Related commands
|
Command |
Description |
|
url-filter log except pre-defined { css | gif | ico | jpg | js | png | swf | xml } |
Disables URL filtering logging for access to resources of a predefined resource type. |
|
session statistics enable |
Enables session statistics collection for software fast forwarding. |
|
display inspect status |
Displays the status of the DPI engine. |
|
dac storage service service-type service-name limit { hold-time time-value | usage usage-value | action { delete | log-only } } |
Sets the storage time limit, storage space usage limit, or the storage limit-triggered action for a service. |
Troubleshooting high CPU usage caused by policy rule matching acceleration
CPU usage is high if object policy rules are modified frequently
Symptom
The system activates rule matching acceleration each time an object policy rule is created or modified, which causes high CPU usage if multiple rules are modified in a short period of time.
Solution
To resolve the issue, upgrade the software to a version that supports delayed rule matching acceleration. This feature enables the system to activate rule matching acceleration for multiple rules together after the rules are completely deployed to prevent frequent acceleration from causing high CPU usage.
Delayed rule matching acceleration for object policies is available in the following software versions:
· D032SP26 and later.
· D045SP07 and later.
High CPU usage caused by low-speed security policy matching
Symptom
Security policies that are not accelerated match packets at a low speed. In this case, multi-policy configuration consumes a large amount of CPU resources.
Solution
To resolve the issue, upgrade the software to a version that supports automatic rule matching acceleration for security policies and enable the automatic acceleration feature. This feature allows the system to activate rule matching acceleration 2 seconds after a policy is created or modified if 100 or fewer policies exist or 20 seconds if over 100 policies exist.
Automatic rule matching acceleration for security polices is available for all D032SP and D045SP software versions.
Troubleshooting RBM and VRRP
Two backups in a VRRP group
Symptom
As shown in Figure 41, configure VRRP groups on two security gateways, which are connected to Layer 2 switches in the uplink and downlink directions. The uplink and downlink interfaces on the devices are operating at Layer 3. However, both security gateways are backup nodes in the VRRP groups.
The following is the configuration:
|
Device |
Configuration |
|
Security gateways |
Set up an RBM channel between the two security gateways. Configure two VRRP groups on the security gateways in the uplink and downlink directions, and associate the VRRP groups with RBM as follows: · Add VRRP groups 1 and 3 on the uplink and downlink interfaces of Device A to the active group. Add VRRP groups 2 and 4 on the uplink and downlink interfaces of Device A to the standby group. · Add VRRP groups 1 and 3 on the uplink and downlink interfaces of Device B to the standby group. Add VRRP groups 2 and 4 on the uplink and downlink interfaces of Device B to the active group. Specify the next hop of the routes on the devices destined to the Internet as the IP address of the router interface connected to the devices (2.1.1.15). |
|
Router |
Specify the next hop for the route destined to Host A as the virtual IP address of VRRP group 1 (2.1.1.3). Specify the next hop for the route destined to Host B as the virtual IP address of VRRP group 2 (2.1.1.4). |
|
Host A |
Specify the default gateway IP address as the virtual IP address of VRRP group 3 (10.1.1.3). |
|
Host B |
Specify the default gateway IP address as the virtual IP address of VRRP group 4 (10.1.1.4). |
|
Switch A |
Add the interfaces connected to the devices and router to the same VLAN. |
|
Switch B |
Add the interfaces connected to the devices and hosts to the same VLAN. |
Solution
To resolve the issue:
1. Execute the display remote-backup-group status command to verify that the RBM control channel connection is normal.
RBM_P[M9012_1]dis remote-backup-group status
Remote backup group information:
Backup mode: Dual-active
Device management role: Primary
Device running status: Active
Data channel interface: Route-Aggregation1023
Local IP: 30.24.0.1
Remote IP: 30.24.0.2 Destination port: 60164
Control channel status: Connected
Keepalive interval: 1s
Keepalive count: 10
Configuration consistency check interval: 1 hour
Configuration consistency check result: Consistent(2020-12-17 10:55:15)
Configuration backup status: Auto sync enabled
Session backup status: Hot backup enabled
Delay-time: 1 min
If the control channel status is connected, the control channel connection status is normal. If the control channel status is disconnected, check the status of the physical interface used by the RBM control channel.
2. Execute the display link-aggregation verbose Blade-Aggregation command to verify that the service module is in Selected status.
RBM_P[M9012_1]dis link-aggregation verbose Blade-Aggregation
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, I -- Individual
Port: A -- Auto port
Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
D -- Synchronization, E -- Collecting, F -- Distributing,
G -- Defaulted, H -- Expired
Aggregate Interface: Blade-Aggregation1
Aggregation Mode: Static
Loadsharing Type: Shar
Port Status Priority Oper-Key
--------------------------------------------------------------------------------
Blade4/0/1 S 32768 4
Blade7/0/1 S 32768 4
Aggregate Interface: Blade-Aggregation257
Aggregation Mode: Static
Loadsharing Type: Shar
Port Status Priority Oper-Key
--------------------------------------------------------------------------------
Blade4/0/2 S 32768 5
Blade7/0/2 S 32768 5
The blade aggregate interface status is S (normal). If the statuses for all blade aggregate interfaces are U, or no blade aggregate interfaces are displayed, check the service engine module status.
3. If the issue persists, contact H3C Support.
Troubleshooting attack detection and prevention failures
FIN flood attack report failure
Symptom
Devices on the external network access the server through the firewall. Attack detection and prevention is configured on the firewall to protect the server from attacks.
However, when external users launch FIN flood attacks on the server, the firewall fails to log FIN flood attack events as configured or forward the traffic.
Figure 42 Network diagram
Attack detection and prevention settings on the firewall:
# Create attack defense policy 1, enable global FIN flood attack detection, and specify global actions against FIN flood attacks.
attack-defense policy 1
fin-flood detect non-specific
fin-flood action logging drop client-verify
# Apply attack defense policy 1 to inbound security zone Untrust.
security-zone name Untrust
attack-defense apply policy 1
Solution
To resolve the issue:
1. Verify that the attack defense policy is applied to the inbound security zone, FIN flood attack detection is enabled, and global actions against FIN flood attacks are specified.
2. Use the display attack-defense malformed-packet statistics command to display statistics about malformed packets and check if malformed packets are dropped.
FIN packets belong to malformed packets.
3. Check if the destination address of the traffic is the same. If yes, check the receiving rate of FIN packets destined for the address. If the rate does not reach the global threshold for triggering FIN flood attack prevention, the firewall does not drop FIN packets. This is normal. If the rate exceeds that threshold, go to the next step.
4. If the issue persists, contact Technical Support.
Related commands
|
Command |
Description |
|
display attack-defense policy {name} |
Displays attack defense policy configuration. |
|
display attack-defense statistics security-zone{ zone } |
Displays dropped attack packet statistics. |
|
display blacklist { ip | ipv6 } |
Displays source IPv4 or IPv6 blacklist entries. |
Troubleshooting threat log generation by IPS
This section provides troubleshooting information for threat log generation by IPS.
No threat logs generated on the IPS device
Symptom
As shown in Figure 43, traffic from or to the PC is forwarded by the switch. The M9012-S device is attached to the switch and performs IPS on the received mirroring traffic.
However, no threat logs have been generated on the IPS device for a long time when attacks are present in the network.
The following settings are configured:
· Configure a mirroring group and mirroring source and destination interfaces.
· Create a blackhole-type bridge instance for inline forwarding, and add interfaces to the instance.
· Configure security zones and add interfaces to the security zones.
· Use an IPS policy in a security policy.
Troubleshooting flowchart
Figure 44 Flowchart for troubleshooting threat log generation failure on the IPS device
Solution
1. Verify that a session is running on the device and the session is normal. You can identify whether the session is normal according to the session status, application, and whether a one-direction flow exists.
Initiator:
Source IP/port: 8:7:6:5:4:3:2:2/6158
Destination IP/port: 1:2:3:4:5:6:7:7/110
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Ten-GigabitEthernet2/2/0/10
Source security zone: Untrust
Responder:
Source IP/port: 1:2:3:4:5:6:7:7/110
Destination IP/port: 8:7:6:5:4:3:2:2/6158
VPN instance/VLAN ID/Inline ID: -/-/-
Protocol: TCP(6)
Inbound interface: Ten-GigabitEthernet2/2/0/9
Source security zone: Trust
State: TCP_ESTABLISHED //If the session state is abnormal, three handshakes cannot finish successfully. In this case, the device cannot perform IPS insepction and generate IPS logs.
Application: POP3 //If the device cannot identify the application, the device cannot generate IPS logs.
Rule ID: 0
Rule name: v6
Start time: 2018-12-27 18:49:14 TTL: 1199s
Initiator->Responder: 5 packets 406 bytes
Responder->Initiator: 4 packets 303 bytes
//If the flow is one-direction flow, IPS inspection fails and the device cannot generate IPS logs.
2. If no session exists, execute the display counters rate inbound interface command to identify whether traffic is mirrored on an interface. If no traffic mirroring is performed, verify that the mirroring settings are correct on the device.
3. Verify that no packet loss occurs by executing the display system internal ip packet-drop statistics and display system internal aspf statistics zone-pair ipv4 commands.
If packets are lost before they reach the DPI module due to configuration errors, the device cannot generate IPS logs.
4. If a session exists but the session is incomplete. Typically, inbound and outbound packets mirrored from the switch do not reach the device through the same physical port or logical interface. In this case, verify that blackhole-type bridge forwarding settings are correct.
5. If the session is normal, perform the following tasks:
a. Identify the license and signature library version.
b. Execute the display security-policy ip command to obtain security policy information. Verify that an IPS policy is used in a security policy and identify security policy hit statistics. Make sure packets are matched with the security policy that is enabled with content security.
c. Execute the display inspect status command to verify that the status of the DPI engine is normal. If the status of the DPI engine is normal is in bypass state, the device does not perform DPI inspection.
display inspect status
Chassis 1 Slot 0:
Running status: normal
d. Execute the display system internal inspect hit-statistics command to verify that the device performs DPI inspection on packets.
display system internal inspect hit-statistics
Rule ID Module Rule hits AC hits PCRE try PCRE hits
1855 IPS 0 1 0 0
The output shows that the device has performed DPI inspection, but the packet hits only the AC part and does not match the whole signature. In this case, the device does not generate a log. If the value for the Rule hits field is not 0, a rule is matched with packets.
Troubleshooting RBM dynamic routing issues
RBM switchover not triggered upon uplink or downlink interface failure
Symptom
Traffic is still sent to an RBM member device for forwarding after its uplink or downlink interface fails.
Solution
To resolve the issue:
1. Log in to the RBM member devices and verify that they have the same number of service modules.
2. Configure the primary member device as follows:
RBM_P[M9016_1-remote-backup-group] track 1 interface Route-Aggregation1
RBM_P[M9016_1-remote-backup-group] track 2 interface Route-Aggregation11
RBM_P[M9016_1-remote-backup-group] display this
#
remote-backup group
backup-mode dual-active
data-channel interface Route-Aggregation1000
delay-time 1
adjust-cost bgp enable absolute 10000
adjust-cost ospf enable absolute 10000
adjust-cost ospfv3 enable absolute 10000
track 1
track 2
local-ip 192.168.195.9
remote-ip 192.168.195.10
device-role primary
3. Configure the secondary member device as follows:
RBM_S[M9016_2-remote-backup-group] track 1 interface Route-Aggregation1
RBM_S[M9016_2-remote-backup-group] track 2 interface Route-Aggregation11
RBM_S[M9016_2-remote-backup-group] display this
#
remote-backup group
backup-mode dual-active
data-channel interface Route-Aggregation1000
delay-time 1
adjust-cost bgp enable absolute 10000
adjust-cost ospf enable absolute 10000
adjust-cost ospfv3 enable absolute 10000
track 1
track 2
local-ip 192.168.195.10
remote-ip 192.168.195.9
device-role secondary
Inconsistent ACL configuration between the RBM member devices
Symptom
The RBM member devices have inconsistent ACL configuration, such as the following:
RBM_P[M9016_1]%Dec 17 14:25:43:191 2020 M9016_1 RBM/6/RBM_CFG_COMPARE_START: Started configuration consistency check.
%Dec 17 14:25:44:775 2020 M9016_1 RBM/6/RBM_CFG_COMPARE_RESULT: The following modules have inconsistent configuration: acl.
%Dec 17 14:25:44:775 2020 M9016_1 RBM/6/RBM_CFG_COMPARE_FINISH: Finished configuration consistency check.
Solution
To resolve the issue:
· If an ACL exists only on the secondary member device, perform the following tasks:
¡ To retain the ACL, create it on the primary member device, and save the running configuration of both RBM member devices.
¡ To delete the ACL, execute the configuration manual-sync command on the primary member device, and save the running configuration of both RBM member devices.
· If an ACL exists only on the primary member device, perform the following tasks:
¡ To retain the ACL, execute the configuration manual-sync command on the primary member device, and save the running configuration of both RBM member devices.
¡ To delete the ACL, delete it from the primary member device, execute the configuration manual-sync command on the primary member device, and save the running configuration of both RBM member devices.
Troubleshooting AFT
IPv6 access to IPv4 network fails
Symptom
This section uses IPv6-to-IPv4 source address dynamic translation and IPv4-to-IPv6 source address static translation as an example.
To allow PC1 to access PC2 in the IPv4 network, the following AFT policies are configured on the M9000 firewall:
· An IPv4-to-IPv6 source address static mapping to map destination IPv4 address 1.1.1.1 to IPv6 address 23::1.
· An IPv6-to-IPv4 source address dynamic translation policy to translate the source addresses in IPv6 packets to IPv4 address 30.30.40.100.
However, AFT fails or the translated packets cannot be forwarded correctly.
Figure 45 Network diagram
Settings on the firewall
acl ipv6 number 2000
rule 0 permit source 1:1::1/128
#
aft address-group 0
address 30.30.40.100 30.30.40.100
#
aft v6tov4 source acl ipv6 number 2000 address-group 0
#
aft v4tov6 source 1.1.1.1 23::1
#
interface Route-Aggregation10.900
aft enable
interface Route-Aggregation10.901
aft enable
Solution
To resolve the issue:
1. Verify that AFT is configured correctly and AFT is enabled on both the input interface and output interface on the M9000 firewall.
[sysname]dis aft configuration
aft address-group 0
address 30.30.40.100 30.30.40.100
aft v6tov4 source acl ipv6 number 2000 address-group 0
aft v4tov6 source 1.1.1.1 23::1
interface Route-Aggregation10.900
aft enable
interface Route-Aggregation10.901
aft enable
AFT ALG:
DNS : Enabled
FTP : Enabled
HTTP : Enabled
ICMP-ERROR : Enabled
RTSP : Enabled
SIP : Enabled
2. Use the debugging aft packet ip command to enable debugging of AFT packets and verify that packets can be translated correctly. If the following information is displayed, AFT packets are translated correctly.
<sysname>debugging aft packet ip
Dec 16 15:08:22:697 2020 H3C AFT/7/COMMON: -Slot=6.1;
PACKET: (Route-Aggregation10.900) Protocol: UDP
1.1.1.1/69 - 30.30.40.100/1128(VPN:0) ------>
23::1/69 – 1:1::1/35017(VPN:0)
Or
<sysname>debugging aft packet ipv6
Dec 16 15:09:13:696 2020 H3C AFT/7/COMMON: -Slot=6.1;
PACKET: (Route-Aggregation10.901) Protocol: UDP
1:1::1/6677 - 23::1/5060(VPN:0) ------>
30.30.40.100/1149 - 1.1.1.1/5060(VPN:0)
3. Verify that the flow tables are deployed normally.
[H3C-probe]dis system internal openflow instance inner-redirect flow-table
Flow entry 3305 information:
cookie: 0x0, priority: 5045, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 900, mask: 0xfff
IP Range: IPv4 destination address from 30.30.40.100 to 30.30.40.100
Instruction information:
Write actions:
Group: 4026531857
Flow entry 3306 information:
cookie: 0x0, priority: 5045, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 4094, mask: 0xfff
IP Range: IPv4 destination address from 30.30.40.100 to 30.30.40.100
Instruction information:
Write actions:
Group: 4026531857
Flow entry 3307 information:
cookie: 0x0, priority: 5080, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
IPv4 source address: 1.1.1.1, mask: 255.255.255.255
Instruction information:
Write actions:
Group: 4026531865
Flow entry 3308 information:
cookie: 0x0, priority: 5085, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
IPv4 destination address: 1.1.1.1, mask: 255.255.255.255
Instruction information:
Write actions:
Group: 4026531865
Flow entry 3309 information:
cookie: 0x0, priority: 7085, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 900, mask: 0xfff
IPv6 destination address: 23::1
IPv6 destination address mask: FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
Instruction information:
Write actions:
Group: 4026531865
Flow entry 3310 information:
cookie: 0x0, priority: 7085, hard time: 0, idle time: 0, flags: check_overlap
|reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --
Match information:
Input interface: RAGG10
VLAN ID: 4094, mask: 0xfff
IPv6 destination address: 23::1
IPv6 destination address mask: FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
Instruction information:
Write actions:
Group: 4026531865
4. If the issue persists, contact Technical Support.
Miscellaneous
Unexpected card reboot because of an internal port failure
Symptom
A card reboots unexpectedly because of an internal port failure.
· As shown in the diagfile.log file, an internal port of the card connected to the peer card went down, and then came up after the module restarted.
<M9k>more diagfile/diagfile.log
%@12527^Dec 19 16:10:56:906 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The source port went down.
%@12528^Dec 19 16:10:56:640 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=13; Chassis 1 Slot 13 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 8: The source port went down.
%@12529^Dec 19 16:10:57:376 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=11; Chassis 1 Slot 11 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 3: The source port went down.
%@12530^Dec 19 16:10:56:740 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=12; Chassis 1 Slot 12 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 6: The source port went down.
%@12554^Dec 19 16:11:11:959 2020 M9k DRV/3/FAULT_MONITOR_BITMAP:
Fault PhySlot List: 3
Fault Reason BitMap:
slot : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
-----------------------------------------------------
Fabric1 : 5 5 5 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fabric2 : 5 5 5 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fabric3 : 5 5 5 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fabric4 : 5 5 5 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5
-----------------------------------------------------
IO board: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fault Reason: 0-RFCS, 1-RERPKT, 2-DOWN, 3-UNRESP, 4-1bit, 5-NORMAL
%@12555^Dec 19 16:11:11:960 2020 M9k DRV/3/FAULT_MONITOR_REBOOT: Chassis 1 Slot 3: The module will be restarted due to a hardware failure.
· As shown in the logfile.log file, an internal port of the card connected to the peer card went down, and then came up after the card is restarted.
<M9k>more logfile/logfile.log
%@4387931%Dec 19 16:10:56:906 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The connectivity of the internal port failed.
%@4387932%Dec 19 16:10:56:640 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=13; Chassis 1 Slot 13 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 8: The connectivity of the internal port failed.
%@4387933%Dec 19 16:10:57:376 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=11; Chassis 1 Slot 11 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 3: The connectivity of the internal port failed.
%@4387934%Dec 19 16:10:56:740 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=12; Chassis 1 Slot 12 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 6: The connectivity of the internal port failed.
%@4387947%Dec 19 16:11:11:960 2020 M9k DRV/3/FAULT_MONITOR_REBOOT: Chassis 1 Slot 3: The module will be restarted due to a hardware failure.
%@4387948%Dec 19 16:11:12:151 2020 M9k DEV/2/BOARD_STATE_FAULT: Board state changed to Fault on chassis 1 slot 3, type is NSQM1FWEFGA0.
Solution
Collect and send the logs to H3C Support for analysis.
Unexpected power-off of a card because of an internal port failure
Symptom
A card powers off unexpectedly because of an internal port failure.
Solution
· As shown in the diagfile.log file, a card rebooted three times within half an hour because of failure of an internal port connected to the peer card, and a message "The module will be isolated due to a hardware failure" is output. This internal port failure cannot be resolved through card reboot and the card will be isolated.
<M9k>more diagfile/diagfile.log
%@12574^Dec 19 17:15:53:091 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The source port went down.
%@12584^Dec 19 17:23:57:002 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The source port went down.
%@12605^Dec 19 17:32:34:001 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The source port went down.
%@12615^Dec 19 17:32:54:996 2020 M9k DRV/3/FAULT_MONITOR_BITMAP:
Fault PhySlot List: 10
Fault Reason BitMap:
slot : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
-----------------------------------------------------
Fabric1 : 5 5 5 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fabric2 : 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fabric3 : 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fabric4 : 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
-----------------------------------------------------
IO board: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
Fault Reason: 0-RFCS, 1-RERPKT, 2-DOWN, 3-UNRESP, 4-1bit, 5-NORMAL
%@12616^Dec 19 17:32:54:996 2020 M9k DRV/3/FAULT_MONITOR_ISOLATE: Chassis 1 Slot 10: The module will be isolated due to a hardware failure.
· As shown in the logfile.log file, a card rebooted three times within half an hour because of failure of an internal port connected to the peer card, and a message "The card will be isolated due to a hardware failure" is output. This internal port failure cannot be resolved through card reboot and the card will be isolated.
<M9k>more logfile/logfile.log
%@4388208%Dec 19 17:15:40:345 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The connectivity of the internal port failed.
%@4388291%Dec 19 17:23:57:002 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The connectivity of the internal port failed.
%@4388385%Dec 19 17:32:34:001 2020 M9k DRV/3/HG_MONITOR_PORT_ERROR: -Chassis=1-Slot=10; Chassis 1 Slot 10 Unit 0 Port 3 to Chassis 1 Slot 3 Unit 0 Port 1: The connectivity of the internal port failed.
%@4388389%Dec 19 17:32:54:996 2020 M9k DRV/3/FAULT_MONITOR_ISOLATE: Chassis 1 Slot 10: The module will be isolated due to a hardware failure.
Solution
Replace the faulty card and send related logs to H3C Support.
Electronic label reading failure
An electronic label is a profile of a device. It contains the permanent configuration, including the name, serial number, MAC address, vendor name, and product code of the device.
The device SN and DID used for license activation file application is contained in the electronic label.
Symptom
The display device manuinf command shows that the electronic label of the device is missing. As a result, the device cannot be licensed because the device SN and DID information cannot be obtained.
Solution
To resolve the issue:
1. View related logs to identify the cause of the issue.
An active/standby MPU switchover might cause loss of the electronic label.
[B-probe]local logbuffer 10 display
Sep 08 2020 16:54:36:488937:
LINE:152-TASK:ofpd-FUNC:BSP_E2PROM_Read_OnSelec:
get I2C MutexSem1 fail.
Sep 08 2020 16:54:36:596761:
LINE:2077-TASK:TEMP-FUNC:drv_sysm_get_power_size_75X:
get I2C MutexSem1 fail.
Sep 08 2020 16:54:37:489907:
LINE:5780-TASK:ofpd-FUNC:DRV_SYSM_SysGetManufactureInfo:
In function:BSP_E2PROM_Read_OnSelec, Read manual infoerror
Sep 08 2020 16:54:37:489967:
LINE:6089-TASK:ofpd-FUNC:DRV_SYSM_ManuInfoResolve:
Read manufacture information Fail!
Sep 08 2020 16:54:37:490005:
LINE:12303-TASK:ofpd-FUNC:DRV_DEVM_GetManuInfo:
get chassis manu info failed!
2. Collect and send related information to H3C Support for help.
MPU and service module version inconsistency
Symptom
A service module is in Fault state because it runs a different software version than the MPU.
You can use the display device or dis version command in any view or the display system internal version command in probe view to view system version information of the device.
Solution
· If the module remains in Fault state and reboots repeatedly, upgrade the version of the module from BootWare.
· If the module is stuck in Fault state, remove and then reinstall the module to reboot it and upgrade the version of the module from BootWare
Related commands
This section lists the commands that you might use for troubleshooting the preceding miscellaneous issues.
|
Command |
Description |
|
display boot-loader |
Displays current software images and startup software images. |
|
display device manuinfo |
Displays electronic label information for the device. |
Troubleshooting load balancing
This section provides troubleshooting information for common load balancing issues.
Health monitoring failure with a TCP probe template
Symptom
A server farm member or a real server is set to the Probe-failed state after the server farm or the real server references a TCP probe template.
Troubleshooting flowchart
Figure 46 Flowchart for troubleshooting health monitoring failure with a TCP probe template
Solution
To resolve the issue:
1. Verify that the port number of the server farm member or real server is not 0.
If the port number is 0, modify the port number to the actual service port.
2. Verify that the IP address of the server farm member or real server can be pinged.
3. Verify that the port number of the server farm member or real server is reachable.
Execute the telnet x.x.x.x port command (for example, telnet 1.1.1.1 8080). If a 400 bad request is returned, the port number is reachable, and proceed to step 4. If nothing is returned, proceed to step 5.
4. Verify that no destination IP address and destination port number are specified in the TCP probe template.
No destination IP address and destination port number are required in the TCP probe template. If you specified the destination IP address and destination port number in the TCP probe template, make sure they are the same as the IP address and port number of the server farm member or real server.
5. Capture packets to troubleshoot the issue. The following situations might exist:
¡ A probe packet is sent, but no response packet is received.
In this example, the IP address of the LB device is 32.253.7.1, the IP address of the server farm member or real server is 10.113.119.1, and the port number of the server farm member or real server is 8080. Captured packets show that the device did not receive a response and retransmitted the probe packet twice. This indicates that it is not the device that causes the health monitoring failure.
For this situation, confirm with the administrator of the server farm member or real server for the following information:
- Whether the server farm member or real server received the probe packet.
- Whether the server farm member or real server replied with a response packet if it received the probe packet.
- Whether the response packet was blocked by other devices on the path.
¡ A probe packet is sent, and an RST packet is received.
In this example, the IP address of the LB device is 32.253.7.1, the IP address of the server farm member or real server is 10.113.119.1, and the port number of the server farm member or real server is 8080.
This situation indicates that the network transmission is normal but the port of the server farm member or real server is unreachable. Contact the administrator to open the port of the server farm member or real server.
¡ A probe packet is sent, and a normal response packet is received.
In this example, the IP address of the LB device is 32.253.6.150, the IP address of the server farm member or real server is 10.113.119.1, and the port number of the server farm member or real server is 8080.
Captured packets show that the device sent an SYN packet, and the server farm member or real server replied with a SYN-ACK packet. Then, the device immediately sent an RST packet to close the connection. This packet exchange process of health monitoring is correct, and the server farm member or real server should not be set to the Probe-failed state.
For this situation, contact H3C Support.
Related commands
This section lists the commands that you might use for troubleshooting health monitoring failures with a TCP probe template.
|
Command |
Description |
|
display server-farm |
Displays server farm information. |
|
display real-server |
Displays real server information. |
|
display current-configuration configuration nqa-tplt-tcp |
Displays TCP health monitoring configuration. |
|
display current-configuration configuration server-farm |
Displays server farm configuration. |
|
display current-configuration configuration real-server |
Displays real server configuration. |
|
debugging tcp packet acl |
Enables TCP packet debugging. |
|
debugging nqa all acl |
Enables NQA packet debugging. |
Health monitoring failure with an HTTP probe template
Symptom
A server farm member or a real server is set to the Probe-failed state after the server farm or the real server references an HTTP probe template.
Troubleshooting flowchart
Figure 47 Flowchart for troubleshooting health monitoring failure with an HTTP probe template
Solution
To resolve the issue:
1. Verify that the port number of the server farm member or real server is not 0.
If the port number is 0, modify the port number to the actual service port.
2. Verify that the IP address of the server farm member or real server can be pinged.
3. Verify that the port number of the server farm member or real server is reachable.
Execute the telnet x.x.x.x port command (for example, telnet 1.1.1.1 8080). If a 400 bad request is returned, the port number is reachable, and proceed to step 4. If nothing is returned, also proceed to step 4.
4. Verify that the HTTP probe packet is padded with correct contents by capturing packets. The following situations might exist:
¡ A response packet is received, and the status code is 4xx.
In this example, the IP address of the LB device is 26.1.1.148, and the IP address of the server farm member or real server is 26.1.1.1. Captured packets show that the header of the probe packet is incorrect.
For this situation, confirm with the administrator for the correct header and reconfigure the header contents.
¡ A response packet is received, and the status code is 200.
In this example, the IP address of the LB device is 26.1.1.148, and the IP address of the server farm member or real server is 26.1.1.1. If the expected status code to determine a successful NQA operation is configured as 200, the health monitoring should succeed.
For this situation, contact H3C Support.
Related commands
This section lists the commands that you might use for troubleshooting health monitoring failures with an HTTP probe template.
|
Command |
Description |
|
display server-farm |
Displays server farm information. |
|
display real-server |
Displays real server information. |
|
display current-configuration configuration nqa-tplt-http |
Displays HTTP health monitoring configuration. |
|
display current-configuration configuration server-farm |
Displays server farm configuration. |
|
display current-configuration configuration real-server |
Displays real server configuration. |
|
debugging http packet acl |
Enables HTTP packet debugging. |
|
debugging nqa all acl |
Enables NQA packet debugging. |
Health monitoring failure with a UDP probe template
Symptom
A server farm member or a real server is set to the Probe-failed state after the server farm or the real server references a UDP probe template.
Troubleshooting flowchart
Figure 48 Flowchart for troubleshooting health monitoring failure with a UDP probe template
Solution
To resolve the issue:
1. Verify that UDP port detection is enabled.
The probe succeeds if the client does not receive any ICMP port unreachable messages within the probe timeout time. If the client receives an ICMP port unreachable message, the probe fails.
You must enable the sending of ICMP destination unreachable messages on the server (use the ip unreachables enable command if the destination device is an H3C device). If the destination device is an H3C device, you must also execute the data-fill or hex-data-fill command on the client with the raw keyword specified. The specified payload fill string can be any value in the value range.
2. Verify that an ICMP probe template is also specified.
A UDP probe template must work with an ICMP probe template to ensure a successful probe. An ICMP probe template is used to test the availability of a link and check whether ICMP error messages can be received on the LB device.
3. Verify that no destination IP address and destination port number are specified in the UDP probe template.
No destination IP address and destination port number are required in the UDP probe template. If you specified the destination IP address and destination port number in the UDP probe template, make sure they are the same as the IP address and port number of the server farm member or real server.
4. Capture packets to identify whether the port number is reachable.
Capture both ping packets and UDP probe packets.
The following situations might exist:
¡ The device receives no response for the UDP probe packet, but receives ping response packets.
In this example, the IP address of the LB device is 26.1.1.148, the IP address of the server farm member or real server is 26.1.1.1, and the port number of the server farm member or real server is 8080. Captured packets show that the device did not receive a response for the UDP probe packet but received ping response packets. In this case, the UDP health monitoring should succeed, and contact H3C Support.
¡ The device receives an ICMP error message for the UDP probe packet, and also receive ping response packets.
In this example, the IP address of the LB device is 26.1.1.148, the IP address of the server farm member or real server is 26.1.1.1.
The UDP health monitoring fails because the device receives an port unreachable error message. In this case, contact the administrator to confirm whether the port number the server farm or the real server is reachable.
Related commands
This section lists the commands that you might use for troubleshooting health monitoring failures with a UDP probe template.
|
Command |
Description |
|
display server-farm |
Displays server farm information. |
|
display real-server |
Displays real server information. |
|
display current-configuration configuration nqa-tplt-udp |
Displays TCP health monitoring configuration. |
|
display current-configuration configuration server-farm |
Displays server farm configuration. |
|
display current-configuration configuration real-server |
Displays real server configuration. |
|
debugging udp packet acl |
Enables TCP packet debugging. |
|
debugging nqa all acl |
Enables NQA packet debugging. |
Failed to import a CA certificate to a PKI domain due to lack of a key file
Symptom
The system prompts that no key file exists when you import a CA certificate provided by the customer to a PKI domain.
Troubleshooting workflow
1. Verify that the CA certificate contains a key file.
2. If the CA certificate contains a key file, separate the key file and import it independently.
3. If the CA certificate does not contain a key file, contact the customer to obtain a key file and import it.
Solution
To resolve the issue:
1. Convert the certificate provided by the customer to non-PEM format: Import the non-PEM certificate file to the browser, and then export it in PEM format.
2. Open the certificate file in text format, and observe whether the certificate file contains the BEGIN RSA PRIVATE KEY field.
3. If the certificate file contains the BEGIN RSA PRIVATE KEY field, the file contains a key file. Paste the content of the BEGIN RSA PRIVATE KEY field, save it as a .key file, and upload the file to the flash memory of the device.
4. Execute the public-key local import rsa key-name filename filename command to import local key pairs.
If the command fails be executed, contact H3C Support.
5. If the certificate file does not contain the BEGIN RSA PRIVATE KEY field, the file does not contains a key file. Contact the customer to obtain the key file. If the customer provides an independent key file, repeat the previous steps to import local key pairs.
Related commands
This section lists the commands that you might use for troubleshooting CA certificate failure due to lack of a key file.
|
Command |
Description |
|
display pki certificate domain domain-name ca |
Displays information about CA certificates in a PKI domain. |
|
display pki certificate domain domain-name local |
Displays information about local certificates in a PKI domain. |
|
display public-key local rsa public |
Displays local public keys. |
Inactive state of a virtual server
Symptom
A virtual server is in inactive state after it is configured according to the configuration example.
Troubleshooting workflow
1. Verify that the virtual server has referenced a server farm or LB policy.
2. Verify that the referenced server farm or the LB policy already exists.
3. Examine the health monitoring state of members in the referenced server farm or of members in the server farm in the referenced LB policy.
4. Verify that redirection is configured for the virtual server or the referenced LB policy.
Solution
To resolve the issue:
1. Verify that the virtual server has referenced a server farm or LB policy.
2. If the virtual server has referenced a server farm or LB policy, verify that the server farm or the LB policy and its server farm already exist.
3. Examine the health monitoring state of the server farm. If the health monitoring state of each member in the server farm is Probe-failed, the state of the virtual server will be inactive. To troubleshoot health monitoring failures, see the first three sections
4. Verify that redirection is configured for the virtual server or the referenced LB policy. If redirection is not configured, the state of the virtual server will be inactive.
5. If the problem persists, contact H3C Support.
Related commands
This section lists the commands that you might use for troubleshooting the inactive state of a virtual server.
|
Command |
Description |
|
display server-farm |
Displays server farm information. |
|
display real-server |
Displays real server information or server farm member information. |
|
display virtual-server |
Displays virtual server information. |
|
display current-configuration configuration server-farm |
Displays server farm configuration. |
|
display current-configuration configuration real-server |
Displays real server configuration. |
|
display current-configuration configuration virtual-server |
Displays virtual server configuration. |
HTTP service access failure for a TCP virtual server
Symptom
HTTP service access fails after the TCP virtual server configuration is completed according to the configuration example.
Troubleshooting workflow
1. Verify that the virtual server is in active state.
2. Verify that the client request traffic has been delivered to the LB device.
3. Verify that the virtual server has both inbound and outbound packet statistics.
4. Capture packets on both the LB device and the server to for comparison and troubleshooting.
Solution
To resolve the issue:
1. Verify that the server is in active state. The virtual server can correctly process services only when it is in active state. If the virtual server is in inactive state, troubleshoot the issue as described in "Inactive state of a virtual server."
2. Verify that the client request traffic has been delivered to the LB device. Execute the display virtual-server statistics command to display virtual server statistics. If matching requests exist, the command output displays associated information in the Total connections field. If the value for the field is 0, a transmission issue might occur on the link between the client and LB device. Troubleshoot the network transmission environment to make sure the traffic can be delivered to the LB device.
3. Verify that both the Received packets and Sent packets fields have statistics.
If only the Received packets field has statistics, the LB device fails to establish a connection with the server. If the health monitoring state is active, the server does not send the response traffic to the LB device. This is because the LB device forwards client requests to the real server without performing source NAT. In this case, configure source NAT on the LB device or contact the administrator of the customer to configure the LB device as the gateway of the server. Additionally, configure routes to make sure the response traffic can reach the LB device.
4. Capture packets on the LB device.
a. Capture packets on the LB device, and the server does not respond.
Example:
The client IP address is 192.168.43.1, the IP address of the virtual server is 6.6.6.6, and the IP address of the real server is 26.1.1.1. After the client sends a request to the virtual server, the LB device forwards it to the real server. Captured packets show that the clients sent three requests and did not receive any responses. Contact the administrator of the customer to confirm whether the server receives the requests and whether the server sends responses if it receives the requests.
b. Capture packets on the LB device, and the server replies with an RST packet.
Example:
The client IP address is 192.168.43.1, the IP address of the virtual server is 6.6.6.6, and the IP address of the real server is 192.168.43.1. After the client sends a request to the virtual server, the LB device forwards it to the real server. After the LB device sends a SYN packet to the real server, the real server replies with an RST packet. Then, the LB device forwards the RST packet to the client to close the connection. In this case, contact the administrator of the customer to confirm why the server replies with an RST packet.
c. Capture packets on the LB device, and the server replies with a 404 packet.
Example:
The client IP address is 192.168.43.1, the IP address of the virtual server is 6.6.6.6, and the IP address of the real server is 192.168.43.1. After the client sends a request to the virtual server, the LB device forwards it to the real server. Then, the real server replies with a 404 packet, and the LB device forwards the 404 packet to the client. In this case, contact the administrator of the customer to confirm why the server replies with a 404 packet.
Related commands
|
Command |
Description |
|
display server-farm |
Displays server farm information. |
|
display real-server |
Displays real server information. |
|
display virtual-server |
Displays virtual server information. |
|
display current-configuration configuration server-farm |
Displays server farm configuration. |
|
display current-configuration configuration real-server |
Displays real server configuration. |
|
display current-configuration configuration virtual-server |
Displays virtual server configuration. |
XFF failure for an HTTP virtual server
Symptom
The X-Forward-For (XFF) function fails to take effect after the configuration is completed according to the configuration example.
Troubleshooting workflow
1. Verify that the configuration is correct and the service traffic matches the behavior in the specified action.
2. Identify whether the client service uses long connection or short connection.
3. Identify whether per-request processing is enabled if the client service uses long connection.
Solution
To resolve the issue:
1. Verify that the XFF behavior is configured for an action to make sure the traffic matches the specified action. If the traffic does not match the action, check the LB class settings and network connectivity.
2. Identify whether the client service uses long connection or short connection.
¡ Short connection—A TCP connection transmits only one HTTP request packet. The connection is closed after the transmission is complete.
¡ Long connection—A TCP connection transmits more than one HTTP request packet. The connection is closed after the transmission is complete.
If no networking issues exist, the action is matched, and the XFF function does not take effect when the short connection is used, contact H3C Support.
3. If no networking issues exist, the action is matched, and the XFF function does not take effect when the long connection is used, specify an HTTP parameter template and configure the header modify per-request command for the template.
If the issue persists, solution H3C Support.
4. View packets when long and short connections are used.
a. View packet information when a short connection is used.
Example:
The TCP connection is a short connection, because it contains only one get request and is closed after the transmission completes. After the XFF processing succeeds, you can see XFF header in the get packet.
b. View packet information when a long connection is used. The XFF function succeeds for the first HTTP request. Consequent requests do not contain the XFF header.
Example:
The TCP connection is a long connection, because it contains two get requests and is closed after the transmission completes.
In the long connection, the XFF function takes effect for the first get request:
The XFF function does not take effect for the second get request:
In this situation, specify an HTTP parameter template and enable per-request processing.
Related commands
|
Command |
Description |
|
display server-farm |
Displays server farm information. |
|
display real-server |
Displays real server information. |
|
display virtual-server |
Displays virtual server information. |
|
display loadbalance class |
Displays LB class information. |
|
display loadbalance action |
Displays LB action information. |
|
display loadbalance policy |
Displays LB policy information. |
|
display current-configuration configuration server-farm |
Displays server farm configuration information. |
|
display current-configuration configuration real-server |
Displays real server configuration information. |
|
display current-configuration configuration virtual-server |
Displays virtual server configuration information. |
Domain name resolution failure for global load balancing
Symptom
The LB device fails to resolve the domain name in a DNS request after you complete configuration according to the configuration example.
Troubleshooting workflow
1. Troubleshoot network connectivity.
2. Check the data center configuration.
3. Check the global DNS listener settings.
4. Check virtual server status.
5. Check links.
6. Check the global virtual server pool status.
7. Check the global DNS mapping.
Solution
To resolve the issue:
1. Troubleshoot network connectivity.
If domain name resolution fails, capture packets to verify that a DNS response is available as follows:
If a DNS response is available and the resolution fails, proceed to step 2.
If no DNS response is available, verify the processing method upon resolution failure configured for the global DNS listener. If the reject method (default) is used, a network connectivity issue exists. Please check the network route between the client and the listener.
If the network connectivity is available and the issue persists, proceed to the next step.
2. Verify that the data center configuration is correct, and the data center has links bound to it and the links are available. To view the link status, go to step 5.
3. Check the global DNS listener settings.
a. Verify that the global DNS listener is enabled, the IP address (of the local host) and other settings are correct. If the global DNS listener is not enabled, the listening service cannot function.
b. Identify the processing method upon resolution failure configured for the global DNS listener.
- If the processing method is reject, view the global DNS listener statistics and capture packets to view DNS packets. In this situation, the statistics contains PJTR (rejected requests) information. The captured information contains DNS request and response packets, and the resolution fails.
View information about rejected requests on the global DNS listener:
[sysname]dis loadbalance global-dns-listener statistics
Chassis 1 Slot 1 CPU 1:
Global DNS listener: gdl1
Received requests: 0
Received valid requests: 0
Unresponded requests: 0
Rejected requests: 0
------------------------------------------------
RCVR - Received requests, RVR - Received valid requests,
UR - Unresponded requests, RJTR - Rejected requests
Type RCVR RVR UR RJTR
A 0 0 0 0
AAAA 0 0 0 0
MX 0 0 0 0
NS 0 0 0 0
CNAME 0 0 0 0
SOA 0 0 0 0
PTR 0 0 0 0
The captured information contains request and response packets, and the resolution fails.
Proceed to step 4 for further troubleshooting.
- If the processing method upon resolution failure is no response, view the global DNS listener statistics and capture packets to view DNS packets. In this situation, the statistics contains UR (unresponded requests) information. The captured information contains only DNS request packets.
View information about unresponded requests on the global DNS listener:
[sysname]dis loadbalance global-dns-listener statistics
Chassis 1 Slot 1 CPU 1:
Global DNS listener: gdl1
Received requests: 0
Received valid requests: 0
Unresponded requests: 0
Rejected requests: 0
------------------------------------------------
RCVR - Received requests, RVR - Received valid requests,
UR - Unresponded requests, RJTR - Rejected requests
Type RCVR RVR UR RJTR
A 0 0 0 0
AAAA 0 0 0 0
MX 0 0 0 0
NS 0 0 0 0
CNAME 0 0 0 0
SOA 0 0 0 0
PTR 0 0 0 0
The captured information contains only request packets.
Proceed to step 4 for further troubleshooting.
4. Verify that the virtual server status is correct. If the virtual server in inactive status, troubleshoot this issue (follow the instructions for troubleshooting real server probe failure). If the virtual server is in active status, proceed to step 5.
5. Verify that the status is active for the links bound to the virtual server pool.
¡ If the probe-failed status is displayed for a link that fails the health monitoring, troubleshoot this issue. It is a general network connectivity issue if the ICMP health monitoring succeeds. You can troubleshoot this issue by pinging the next hop IP address. If the health monitoring succeeds and resolution still fails, proceed to step 6.
¡ If the link status is active or unknown (when health monitoring is not configured), proceed to step 6.
[sysname]display loadbalance link brief
Link Router IP/Interface State VPN instance Link group
glb1_link1 10.10.0.254 Active
6. Verify that the virtual server and links are bound to the virtual server pool correctly, and the health monitoring succeeds for the virtual server pool. Troubleshoot the heal monitoring failure, if any. If the health monitoring succeeds, proceed to step 7 for further troubleshooting.
View the virtual server status in the virtual server pool. The status is active if the health monitoring succeeds, and is inactive if the health monitoring fails.
[sysname]display loadbalance global-virtual-server-pool name glb1
Global virtual server pool: glb1_netconf
Predictor:
Preferred: RR
Alternate: --
Fallback: --
Bandwidth busy-protection: Disabled
Total virtual servers: 1
Active virtual servers: 1
Data center: dc1_gongwang
Server: glb1_slb
Virtual server list:
Name State Address Port Weight Link
netconf_vip101_1 Active 30.0.101.1 80 100 glb1_link1
7. Verify that the global DNS mapping service is enabled, and the global virtual server pool is specified correctly. If the feature is not enabled and the virtual server pool is not correctly specified, the status of the virtual server in the virtual server pool is displayed as inactive, and DNS response packets cannot be returned.
View the global DNS mapping to verify the service status, global virtual server pool, and domain name information are correct:
[sysname]display loadbalance global-dns-map
Global DNS mapping: gdm
Service state: Enabled
TTL: 3600s
Predictor:
Preferred: round-robin
Alternate: --
Fallback: --
Domain name list: www.glb.com
Global virtual server pool list:
Name Weight
gvsp 100
In conclusion, the entire global load balancing process requires correct user configuration (use the configuration example as a reference, especially the restrictions and guidelines section). In addition, make sure the relevant objects are specified correctly, and network connectivity is available. If the issue persists, contact H3C Support.
Related commands
|
Command |
Description |
|
display loadbalance data-center |
Displays data center information. |
|
display loadbalance data-center link statistics |
Displays outbound link statistics for a data center. |
|
display loadbalance default-syncgroup member |
Displays default synchronization group member information. |
|
display loadbalance global-dns-listener |
Displays global DNS listener information. |
|
display loadbalance global-dns-listener statistics |
Displays global DNS listener statistics. |
|
display loadbalance global-dns-map |
Displays global DNS mapping information. |
|
display loadbalance global-dns-map statistics |
Displays global DNS mapping statistics. |
|
display loadbalance global-virtual-server-pool |
Displays virtual server pool information. |
|
display loadbalance global-virtual-server-pool probe |
Displays health monitoring information for a virtual server or virtual IP address. |
|
display loadbalance link |
Displays link information. |
|
reset loadbalance global-dns-listener statistics |
Clears global DNS listener statistics. |
|
reset loadbalance global-dns-map statistics |
Clears global DNS mapping statistics. |
Outbound link load balancing failure
Symptom
Service traffic are not forwarded through the outbound link configured according to the configuration example.
Troubleshooting workflow
1. Verify that the link health monitoring status is correct.
2. Verify that the traffic matches the specified LB policy.
3. Check for conflicting LB policy settings.
Solution
To resolve the issue:
1. Verify that the link health monitoring status is correct. For more information, see the troubleshooting methods in relevant sections of this document.
2. Verify that the traffic can match the specified LB policy. Only matching traffic can be forwarded through the specified link.
You can use the Test LB Configuration function on the Web interface to facilitate verification.
Example:
Specify the parameters on the page as follows:
¡ Destination IP address/Source IP Address: Enter the real IP destination and source addresses.
¡ Destination Port/Source Port: You can specify any destination and source port numbers for outbound link load balancing.
¡ Protocol Layer: Select Layer 4 (required).
¡ Protocol Name: Select a protocol as needed.
After clicking Test, you can see the specified LB policy. If the test result does not match the LB policy, troubleshoot the configuration.
If the test result matches the LB policy, but the traffic is not forwarded through the specified link, contact H3C Support.
3. Check for conflicting LB policy settings.
In the outbound link load balancing scenario, the LB device assigns links to outbound traffic (from the internal network to the external network) that typically matches the specified destination IP address (ISP address). In addition, the LB device assigns links to inbound traffic (from the external network to the internal network) that matches the specified source or destination IP address. If multiple LB policy settings conflict, the inbound and outbound traffic might match incorrect rules and forwarded through the unexpected links. If the issue persists when no such conflicts exist or traffic can correctly match the LB policy even though such a conflict exists, contact H3C Support.
Related commands
|
Command |
Description |
|
display loadbalance link |
Displays link information. |
|
display loadbalance link-group |
Displays link group information. |
|
display virtual-server |
Displays virtual server information. |
|
display loadbalance class |
Displays LB class information. |
|
display loadbalance action |
Displays LB action information. |
|
display loadbalance policy |
Displays LB policy information. |
|
display current-configuration configuration link |
Displays link configuration information. |
|
display current-configuration configuration link-group |
Displays link group configuration information. |
|
display current-configuration configuration virtual-server |
Displays virtual server configuration information. |
RBM channel establishment failure
Symptom
After you configure the remote-backup group command on both devices, view the RBM channel status. The control channel status is Disconnected, indicating the RBM channel cannot be established. The primary device cannot synchronize service entries or configuration to the standby device.
Troubleshooting workflow
1. Verify that the versions of the devices are consistent.
2. Use the display remote-backup-group status command to view information including RBM channel status.
3. Identify the local IP address, peer IP address, and destination port.
4. Identify the device role in the RBM group.
5. Identify the IP addresses of the associated interfaces.
6. Check the link status and protocol status of the associated interfaces.
7. Identify whether the RBM channel is a Layer 2 or Layer 3 connection.
8. View connection log information about the RBM channel for troubleshooting and analysis.
Solution
To resolve the issue:
1. Identify the versions of the primary and secondary devices.
Make sure both devices run the same version.
2. Identify the RBM status.
Use the display remote-backup-group status command to view the control channel status. If the status is Connected, the RBM channel is operating correctly. If the status is Disconnected, the RBM channel is disconnected. You need to locate the reason for the RBM channel establishment failure.
Example:
RBM_P<sysname>display remote-backup-group status
Remote backup group information:
Backup mode: Active/standby
Device management role: Primary
Device running status: Active
Data channel interface: Route-Aggregation1
Local IP: 1.1.1.1
Remote IP: 1.1.1.2 Destination port: 60064
Control channel status: Disconnected
Keepalive interval: 1s
Keepalive count: 10
Configuration consistency check interval: 24 hour
Configuration consistency check result: Not Performed
Configuration backup status: Auto sync enabled
Session backup status: Hot backup enabled
Uptime since last switchover: 0 days, 0 hours, 12 minutes
3. Identify the local IP address, peer IP address, and destination port.
Use the display remote-backup-group status command to identify whether the local and peer IP addresses are configured for both the primary and secondary devices in the RBM group. In addition, verify that the local IP address configured on the local device matches the peer IP address configured on the peer device, and the peer IP addresses can be pinged on both devices. Then, verify that the destination ports configured on both devices are consistent for successful RBM channel establishment.
4. Identify the device role in the RBM group.
Verify that the correct roles are assigned to the devices. In the output from the display remote-backup-group status command, if Device management role is not available, and the value for Device running status is Initial, no roles are assigned to the devices. As a result, the RBM channel cannot be established.
<sysname>dis remote-backup-group status
Remote backup group information:
Backup mode: Active/standby
Device running status: Initial
Data channel interface: Route-Aggregation1
Local IP: 1.1.1.1
Remote IP: 1.1.1.2 Destination port: 60064
Control channel status: Disconnected
Keepalive interval: 1s
Keepalive count: 10
Configuration consistency check interval: 24 hour
Configuration consistency check result: Not Performed
Configuration backup status: Auto sync enabled
Session backup status: Hot backup enabled
Uptime since last switchover: 0 days, 0 hours, 0 minutes
Use the device-role { primary | secondary } command to assign the primary and secondary roles to the devices in the RBM group. Make sure the roles of the two devices are different.
5. Identify the IP addresses of the associated interfaces.
Check whether the interfaces associated with the local IP addresses (Local-ip) exist on the primary and secondary devices. Then execute the display interface brief command to verify that corresponding IP addresses are configured for the interfaces on the devices. If no such IP addresses are available, configure the IP addresses identified by Local-ip for the RBM channel interfaces on the primary and secondary devices.
RBM_P<sysname >display interface brief
Brief information on interfaces in route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Primary IP Description
FGE1/2/3/9 UP UP 1.1.1.1
When the local and peer IP addresses can be pinged and the associated interfaces are not assigned to any security zones or specified for any security policies, if the control channel status is still Disconnected, contact H3C Support.
6. Check the link status and protocol status of the associated interfaces.
Use the display interface brief command to check whether the interfaces are physical or aggregate interfaces, and whether their link status and protocol status are UP.
RBM_P<sysname >display interface brief
Brief information on interfaces in route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Primary IP Description
FGE1/2/3/9 UP UP 1.1.1.1
If the RBM channel interfaces are physical interfaces in DOWN state, verify that the transceiver modules, optical fibers, modes, rates, and hardware are correct for the interfaces on the local and peer devices.
If the RBM channel interfaces are aggregate interfaces in DOWN state, check whether the aggregate interfaces have member ports and the member ports are in UP state. Then verify that the link aggregation modes (static or dynamic) are consistent for the aggregate interfaces on the local and peer devices.
7. Identify whether the RBM channel is a Layer 2 or Layer 3 connection.
¡ Layer 2 connection:
If the RBM channel is a Layer 2 connection (the local and peer IP address are on the same subnet), check whether the RBM channel is directly connected or connected at Layer 2 through a switch.
If the RBM channel is directly connected and the local and peer IP addresses cannot ping each other, contact H3C Support.
If the RBM channel is connected at Layer 2 through a switch, verify that the interfaces on the devices connected to the switch operate at Layer 2 and are assigned to the same VLAN. In addition, verify that VLAN interfaces are configured on the switch and assigned the IP addresses on the same subnets as the local and peer IP addresses. The local and peer IP addresses can be pinged. If they cannot be pinged, contact H3C Support.
¡ Layer 3 connection:
If the RBM channel is a Layer 3 connection (the local and peer IP address are on different subnets), and the peer IP address cannot be pinged, use the display ip routing-table command to verify that the primary and secondary devices each have a route to the peer IP address. If the routes are not available, configure the routes as required. Verify that you can ping the IP addresses of the devices on the switch. If the switch is operating correctly, and it cannot ping the device IP addresses, contact H3C Support.
RBM_P<sysname>display remote-backup-group status
Remote backup group information:
Backup mode: Active/standby
Device management role: Primary
Device running status: Active
Data channel interface: Route-Aggregation1
Local IP: 1.1.1.1
Remote IP: 2.2.2.1 Destination port: 60064
Control channel status: Connected
Keepalive interval: 1s
Keepalive count: 10
Configuration consistency check interval: 24 hour
Configuration consistency check result: Not Performed
Configuration backup status: Auto sync enabled
Session backup status: Hot backup enabled
Uptime since last switchover: 0 days, 0 hours, 4 minutes
RBM_P<sysname>display ip routing-table 2.2.2.1
Summary count : 1
Destination/Mask Proto Pre Cost NextHop Interface
2.2.2.1/32 Static 60 0 1.1.1.2 FGE1/2/3/9
8. View connection log information about the RBM channel.
If the RBM channel is disconnected and cannot be reconnected during stable operation of the device, verify that the following log information exists in the log buffer. Identify the connection status of the RBM channel, and check whether the RBM channel flaps repeatedly for further analysis.
¡ The RBM channel is successfully established:
%Jan 19 10:40:02:951 2022 sysname RBM/1/RBM_KEEPALIVE: Local IP=1.1.1.1, remote IP=1.1.1.2, status=Connected
¡ The RBM channel failed to be established:
%Jan 19 10:42:29:172 2022 sysname RBM/1/RBM_KEEPALIVE: Local IP=1.1.1.1, remote IP=1.1.1.2, status=Disconnected
Related commands
|
Command |
Description |
|
display interface brief |
Displays brief interface information. |
|
display ip routing-table |
Displays routing table information. |
|
display remote-backup-group status |
Displays RBM group status information. |
