Troubleshooting interfaces

Tunnel interface issues

Tunnel interface instability

Symptom

After you configure a P2P tunnel (for example, an IPv4, GRE, or IPv6 tunnel), the local tunnel interface is in up state. You can ping the IP address of the remote tunnel interface from the local tunnel interface. However, the local tunnel interface is unstable. The following symptoms exist:

·     The tunnel interface is repeatedly coming up and going down.

·     The tunneled packet loss rate is high and the transmission rate is low.

This section uses a GRE/IPv4 tunnel as an example to describe the troubleshooting procedure.

Common causes

The following are the common causes of this type of issue:

·     Routes destined for the tunnel destination address are flapping, which causes the tunnel to be flapping.

·     The same source and destination addresses are configured on the device for two tunnels. As a result, only one tunnel can come up.

·     Keepalive is enabled on the GRE tunnel interface. However, the device cannot correctly send or receive GRE keepalive packets. As a result, the device places the tunnel in down state.

·     The device does not have sufficient resources to successfully issue the tunnel to the hardware. As a result, the tunnel is down on the physical layer.

·     The configuration on the tunnel interface is inappropriate, leading to the loss of tunneled packets.

Troubleshooting flow

Figure 1 shows the troubleshooting flowchart.

Figure 1 Flowchart for troubleshooting tunnel interface instability

‌

Solution

1.     Examine whether routes are flapping.

Execute the debugging tunnel event command to enable tunneling event debugging. If the system continuously generates route refresh or deletion messages, routes are flapping. In this case, the tunnel is also flapping. The following information shows a sample command output:

<Sysname> debugging tunnel event

<Sysname> %Jun 16 12:49:55:497 2022 Sysname BGP/5/BGP_STATE_CHANGED: BGP.: 4.4.4.4  state has changed from ESTABLISHED to IDLE for TCP_Connection_Failed event received.

//The system received a TCP connection failure event from the BGP peer at 4.4.4.4. The state of the BGP session has changed from Established to Idle.

%Jun 16 12:49:55:497 2022 Sysname BGP/5/BGP_STATE_CHANGED_REASON: BGP.: 4.4.4.4  state has changed from ESTABLISHED to IDLE. (Reason: TCP connection failed(No route to host))

//The BGP session established for the BGP peer at 4.4.4.4 has changed from Established state to Idle state due to a TCP connection failure (no route to reach the host).

If routes are flapping, locate the cause of the route flapping based on the route refresh or deletion messages. For example, if the BGP session cannot stably enter the Established state, troubleshoot the issue according to the BGP troubleshooting manual.

If routes are not flapping, proceed to step 2.

2.     Identify where the device at each end has tunnels with the same source and destination addresses.

Execute the display interface tunnel command in any view on the device at each end of the tunnel and identify whether the same device has multiple tunnels that use the same source and destination addresses.

<Sysname> display interface Tunnel

Tunnel0

Current state: UP

Line protocol state: UP

Description: Tunnel0 Interface

Bandwidth: 64 kbps

Maximum transmission unit: 1464

Internet protocol processing: Disabled

Output queue - Urgent queuing: Size/Length/Discards 0/100/0

Output queue - Protocol queuing: Size/Length/Discards 0/500/0

Output queue - FIFO queuing: Size/Length/Discards 0/75/0

Last clearing of counters: 15:20:18 Mon 06/13/2022

Tunnel source 1.1.1.1, destination 2.2.2.2

...

If multiple tunnels use the same source and destination addresses on the same device, only one of them can come up. You can execute the undo interface tunnel command to delete unneeded tunnels. If the same device does not have multiple tunnels that use the same source and destination addresses, proceed to step 3.

3.     Identify whether GRE keepalive is configured and whether GRE keepalive packets can be sent and received correctly.

Execute the display current interface tunnel command in any view to display the keepalive configuration of the tunnel interface.

<Sysname> display current interface tunnel

#

interface Tunnel2 mode gre

 ip address 10.1.1.2 255.255.255.0

 source 12.1.1.4

 destination 12.1.1.2

 keepalive 3 3

#

On the local end, execute the debugging gre packet command to enable GRE packet debugging to identify whether the local end can correctly receive and send keepalive packets.

<Sysname> debugging gre packet

*Jun 16 12:46:50:350 2022 Sysname GRE/7/packet:

 Tunnel2 packet: Before encapsulation,

   12.1.1.2->12.1.1.4 (length = 24)

*Jun 16 12:46:50:350 2022 Sysname GRE/7/packet:

 Tunnel2 packet: After encapsulation,

   12.1.1.4->12.1.1.2 (length = 48)

*Jun 16 12:46:50:351 2022 Sysname GRE/7/packet:

 Tunnel2 packet: Before de-encapsulation according to fast-forwarding table,

   12.1.1.2->12.1.1.4 (length = 24)

*Jun 16 12:46:50:351 2022 Sysname GRE/7/packet:

 Tunnel2 : Received a keepalive packet.

//Tunnel 2 received a keepalive packet.

On the remote end, enable GRE packet debugging. If the remote end has sent keepalive packets but the local end does not receive any of them, the GRE keepalive packets might fail to pass the local checksum check. As a result, the local tunnel interface goes down. You can execute the undo gre checksum command to disable GRE checksum on the local end or execute the undo keepalive command to disable GRE keepalive.

If the local end can successfully receive keepalive packets from the remote end, proceed to step 4.

4.     Verify that the hop limit or TTL and DF bit settings of tunneled packets are properly configured.

Execute the display current interface tunnel command in any view to check the configuration of the hop limit or TTL and DF bit parameters.

#

interface Tunnel2 mode gre

 ip address 10.1.1.2 255.255.255.0

 source 12.1.1.4

 destination 12.1.1.2

 keepalive 3 3

 tunnel ttl 1

tunnel dfbit enable

#

If the parameters are not properly configured, tunneled packets might be discarded.

A too small hop limit or TTL value might cause tunnel packets to be discarded on intermediate devices due to TTL timeout. In this case, execute the tunnel ttl command in tunnel interface view to set a proper TTL value according to the actual network configuration.

If the DF bit is set for tunneled packets, intermediate devices might discard tunneled packets if the length of these packets exceeds the MTU of the interfaces on the forwarding path. In this case, set the MTU of each interface on the forwarding path to be greater than the length of tunneled packets. If you cannot ensure that the MTU of each interface on the forwarding path is greater than the length of tunneled packets, do not set the DF bit for tunneled packets.

If the issue persists, proceed to step 5.

5.     Identify whether the system fails to process the event for issuing the tunnel to hardware.

Enable tunneling event debugging, and observe whether the system has tunneled packets or events that have failed to be issued to the kernel or driver. The following information shows an example:

<Sysname>debugging tunnel all

*Jun 16 12:51:25:832 2022 Sysname TUNNEL/7/event:

 Tunnel2 notifies driver: Operation = 4.

 TunnelIfIndex = 524, EvilinkIfIndex = 0

 VRFIndex = 0, DstVRFIndex = 0

 TunnelMode = IPv4 GRE, TransPro = 1

 TunnelSrc = 12.1.1.4

 TunnelDst = 12.1.1.2

 TTL = 255, ToS = 0, DFBit = 0

 MTU = 1476, IPv6Mtu = 1476

 DrvContext[0] = 0xffffffffffffffff, DrvContext[1] = 0xffffffffffffffff

 VNHandle = 0x20000040, ADJIndex = 0xfaf3889c

//Tunnel interface 2 notified the driver to execute operation 4.

*Jun 16 12:51:25:832 2022 Sysname TUNNEL/7/event:

 Processing result of operation 4 for Tunnel2: failed.

//The driver failed to process operation 4 issued by tunnel interface 2.

%Jun 16 12:51:25:832 2022 Sysname IFNET/3/PHY_UPDOWN: Physical state on the interface Tunnel2 changed to down.

%Jun 16 12:51:25:832 2022 Sysname IFNET/5/LINK_UPDOWN: Line protocol state on the interface Tunnel2 changed to down.

//Tunnel interface 2 went down.

*Jun 16 12:51:27:350 2022 Sysname TUNNEL/7/event:

 Tunnel2 can't come up because there is not enough hardware resource

//Tunnel 2 cannot come up because of insufficient hardware resources.

If the device generates the event or error messages in Table 1, a hardware fault causes tunnel instability. In this case, contact Technical Support.

Table 1 Debugging messages related to hardware

Field	Description
Tunnelnum can't come up because reason.	Reason why a tunnel interface cannot come up. The value for the reason variable is there is not enough hardware resource.
Failed to save 6RD prefix to DBM.	The system failed to save the IPv6 prefix of the 6RD tunnel to the database in memory (DBM).
Failed to save IPv4 prefix/suffix for 6RD tunnel to DBM.	The system failed to save the IPv4 prefix or suffix of the 6RD tunnel to the DBM.
Failed to save 6RD BR address to DBM.	The system failed to save the BR address of the 6RD tunnel to the DBM.
Failed to send 6RD prefix to kernel.	The system failed to send the 6RD prefix configuration message to the kernel for the tunnel.
Failed to send IPv4 prefix/suffix for 6RD tunnel to kernel.	The system failed to send the 6RD IPv4 configuration message to the kernel for the tunnel.
Failed to send 6RD BR address to kernel.	The system failed to send the 6RD BR address configuration message to the kernel for the tunnel.

 

6.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A