Troubleshooting EVPN

Troubleshooting EVPN VXLAN

Intra-VXLAN tunnel setup failure in an EVPN VXLAN network with distributed gateways

Symptom

In an EVPN network with distributed gateways, tunnels cannot be established between VTEPs in the same VXLAN.

Common causes

The following are the common causes of this type of issue:

·     Type-2 EVPN routes (MAC/IP advertisement routes) and type-3 EVPN routes (IMET routes) have not been received.

·     The RT configuration is incorrect on EVPN instances.

Troubleshooting flow

Troubleshoot the issue by using the following process:

1.     Verify that type-2 routes have been received.

2.     Verify that type-3 routes have been received.

3.     Verify that the RT configuration for EVPN instances is correct.

Figure 1 shows the troubleshooting process.

Figure 1 Flowchart for troubleshooting intra-VXLAN tunnel setup failure

 

Solution

To resolve the issue:

1.     Execute the display bgp l2vpn evpn command on the local end to check if the local end has advertised type-2 or type-3 routes to the peer end. For example, the following output indicates that the local end has advertised type-2 and type-3 routes to 4.4.4.4. If a Route Reflector (RR) exists in the network, specify the RR address when you execute the display bgp l2vpn evpn command. If not, specify the peer end's address.

<Sysname> display bgp l2vpn evpn peer 4.4.4.4 advertised-routes

 

 Total number of routes: 2

 

 BGP local router ID is 1.1.1.1

 Status codes: * - valid, > - best, d - dampened, h - history,

               s - suppressed, S - stale, i - internal, e - external,

               a - additional-path

               Origin: i - IGP, e - EGP, ? - incomplete

 

 Route distinguisher: 2:2

 Total number of routes: 2

 

     Network            NextHop         MED        LocPrf             Path/Ogn

 

* >  [2][0][48][0e86-19b6-0308][0][0.0.0.0]/104

                        0.0.0.0         0          100                i

* >  [3][0][32][1.1.1.1]/80

                        0.0.0.0         0          100                i

¡     If the local end has advertised type-2 or type-3 routes to the peer end, go to step 2.

¡     If the local end has not advertised type-2 and type-3 routes to the peer end, check if BGP is configured correctly for EVPN. For more information, see EVPN VXLAN configuration in EVPN Configuration Guide.

2.     Execute the display bgp l2vpn evpn command on the peer end to check if the peer end has advertised type-2 or type-3 routes to the local end. For example, the following output indicates that the peer end has advertised type-2 and type-3 routes to 4.4.4.4. If an RR exists in the network, specify the RR address when you execute the display bgp l2vpn evpn command. If not, specify the peer end's address.

<Sysname> display bgp l2vpn evpn peer 4.4.4.4 advertised-routes

 

 Total number of routes: 2

 

 BGP local router ID is 3.3.3.3

 Status codes: * - valid, > - best, d - dampened, h - history,

               s - suppressed, S - stale, i - internal, e - external,

               a - additional-path

               Origin: i - IGP, e - EGP, ? - incomplete

 

 Route distinguisher: 1:1

 Total number of routes: 2

 

     Network            NextHop         MED        LocPrf             Path/Ogn

 

* >  [2][0][48][0e86-23cf-0507][0][0.0.0.0]/104

                        0.0.0.0         0          100                i

* >  [3][0][32][3.3.3.3]/80

                        0.0.0.0         0          100                i

¡     If the peer end has advertised type-2 or type-3 routes to the local end, go to step 3.

¡     If the peer end has not advertised type-2 and type-3 routes to the local end, check if BGP is configured correctly for EVPN. For more information, see EVPN VXLAN configuration in EVPN Configuration Guide.

3.     Execute the display this command in VSI view to check if export targets and import targets on both ends are correct.

[Sysname-vsi-aaa] display this

#

vsi aaa

 vxlan 10

 evpn encapsulation vxlan

  route-distinguisher 2:2

  vpn-target 1:1 export-extcommunity

  vpn-target 2:2 import-extcommunity

#

return

¡     If the route targets are inconsistent on the local and peer ends, execute the vpn-target command in VSI view to modify the incorrect route targets.

¡     If the route targets are consistent on the local peer and ends, go to step 4.

4.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

¡     Diagnostic information collected by using the display diagnostic-information command.

Related alarm and log messages

Alarm messages

None.

Log messages

None.

Inter-VXLAN tunnel setup failure in an EVPN VXLAN network with distributed gateways

Symptom

In an EVPN network with distributed gateways, tunnels cannot be established between VTEPs in different VXLANs.

Common causes

The following are the common causes of this type of issue:

·     Type-2 EVPN routes and type-5 EVPN routes (IP prefix advertisement routes) have not been received.

·     The RT configuration is incorrect on VPN instances.

Troubleshooting flow

Troubleshoot the issue by using the following process:

1.     Verify that type-2 routes have been received.

2.     Verify that type-5 routes have been received.

3.     Verify that the RT configuration for VPN instances is correct.

Figure 2 shows the troubleshooting process.

Figure 2 Flowchart for troubleshooting inter-VXLAN tunnel setup failure

 

Solution

To resolve the issue:

1.     Execute the display bgp l2vpn evpn command on the local end to check if the local end has advertised type-2 or type-5 routes to the peer end. For example, the following output indicates that the local end has advertised type-2 and type-5 routes to 4.4.4.4. If an RR exists in the network, specify the RR address when you execute the display bgp l2vpn evpn command. If not, specify the peer end's address.

<Sysname> display bgp l2vpn evpn peer 4.4.4.4 advertised-routes

 

 Total number of routes: 3

 

 BGP local router ID is 1.1.1.1

 Status codes: * - valid, > - best, d - dampened, h - history,

               s - suppressed, S - stale, i - internal, e - external,

               a - additional-path

               Origin: i - IGP, e - EGP, ? - incomplete

 

 Route distinguisher: 1:1

 Total number of routes: 1

 

     Network            NextHop         MED        LocPrf             Path/Ogn

 

* >  [5][0][24][10.1.1.0]/80

                        0.0.0.0         0          100                i

 

 Route distinguisher: 2:2

 Total number of routes: 2

 

     Network            NextHop         MED        LocPrf             Path/Ogn

 

* >  [2][0][48][0e86-19b6-0308][0][0.0.0.0]/104

                        0.0.0.0         0          100                i

* >  [3][0][32][1.1.1.1]/80

                        0.0.0.0         0          100                i

¡     If the local end has advertised type-2 or type-5 routes to the peer end, go to step 2.

¡     If the local end has not advertised type-2 and type-5 routes to the peer end, check if BGP is configured correctly for EVPN. For more information, see EVPN VXLAN configuration in EVPN Configuration Guide.

2.     Execute the display bgp l2vpn evpn command on the peer end to check if the peer end has advertised type-2 or type-5 routes to the local end. For example, the following output indicates that the peer end has advertised type-2 and type-5 routes to 4.4.4.4. If an RR exists in the network, specify the RR address when you execute the display bgp l2vpn evpn command. If not, specify the peer end's address.

<Sysname> display bgp l2vpn evpn peer 4.4.4.4 advertised-routes

 

 Total number of routes: 3

 

 BGP local router ID is 3.3.3.3

 Status codes: * - valid, > - best, d - dampened, h - history,

               s - suppressed, S - stale, i - internal, e - external,

               a - additional-path

               Origin: i - IGP, e - EGP, ? - incomplete

 

 Route distinguisher: 1:1

 Total number of routes: 2

 

     Network            NextHop         MED        LocPrf             Path/Ogn

 

* >  [2][0][48][0e86-23cf-0507][0][0.0.0.0]/104

                        0.0.0.0         0          100                i

* >  [3][0][32][3.3.3.3]/80

                        0.0.0.0         0          100                i

 

 Route distinguisher: 3:3

 Total number of routes: 2

 

     Network            NextHop         MED        LocPrf             Path/Ogn

 

* >  [5][0][24][10.1.1.0]/80

                        0.0.0.0         0          100                i

¡     If the peer end has advertised type-2 or type-5 routes to the local end, go to step 3.

¡     If the peer end has not advertised type-2 and type-5 routes to the local end, check if BGP is configured correctly for EVPN. For more information, see EVPN VXLAN configuration in EVPN Configuration Guide.

3.     Execute the display this command in L3VNI VPN instance view to check if export targets and import targets on both ends are correct.

[Sysname-vpn-instance-vpna] display this

#

ip vpn-instance vpna

 route-distinguisher 1:1

 #

 address-family evpn

  vpn-target 1:1 import-extcommunity

  vpn-target 1:1 export-extcommunity

#

return

¡     If the route targets are inconsistent on the local and peer ends, execute the vpn-target command to modify the incorrect route targets.

¡     If the route targets are consistent on the local peer and ends, go to step 4.

4.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

¡     Diagnostic information collected by using the display diagnostic-information command.

Related alarm and log messages

Alarm messages

None.

Log messages

None.

Layer 2 VXLAN business traffic interruption

Symptom

A VXLAN network cannot forward Layer 2 VXLAN business traffic.

Common causes

The following are the common causes of this type of issue:

·     ACs or VXLAN tunnels are not established.

·     MAC addresses are not learned.

Troubleshooting flow

Figure 3 shows the troubleshooting flowchart.

Figure 3 Flowchart for troubleshooting Layer 2 VXLAN business traffic interruption

 

Solution

To resolve the issue:

1.     Execute the display l2vpn vsi verbose command to view the VXLAN tunnels and ACs of the involved VSI.

<Sysname> display l2vpn vsi verbose

VSI Name: vpna

  VSI Index               : 0

  VSI State               : Up

  MTU                     : 1500

  Bandwidth               : Unlimited

  Broadcast Restrain      : Unlimited

  Multicast Restrain      : Unlimited

  Unknown Unicast Restrain: Unlimited

  MAC Learning            : Enabled

  MAC Table Limit         : -

  MAC Learning rate       : -

  Drop Unknown            : -

  Flooding                : Enabled

  Statistics              : Disabled

  VXLAN ID                : 10

  Tunnels:

    Tunnel Name          Link ID    State  Type      Flood proxy

    Tunnel1              0x5000001  Up     Manual    Disabled

  ACs:

    AC                               Link ID    State    Type

    GE0/0/1 srv1000                  0          Up       Manual

¡     If both the ACs and VXLAN tunnels are up, go to step 2.

¡     If an AC is down, modify the incorrect AC configuration.

¡     If a VXLAN tunnel is down, troubleshoot the issue as described in "Intra-VXLAN tunnel setup failure in an EVPN VXLAN network with distributed gateways."

2.     Execute the display l2vpn mac-address command to check the VSI MAC address table for the MAC addresses of endpoints in the network and the total number of learned MAC address entries.

<Sysname> display l2vpn mac-address

* - The output interface is issued to another VSI

MAC Address    State     VSI Name                        Link ID/Name   Aging

0001-0001-0001 Static    aaa                             Tunnel1        NotAging

52f6-bc1e-0d06 Dynamic   vpna                            GE0/0/1        Aging

--- 3 mac address(es) found  ---

¡     After you collect the preceding information, proceed with step 3.

3.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

¡     Diagnostic information collected by using the display diagnostic-information command.

Related alarm and log messages

Alarm messages

None.

Log messages

None.

Layer 3 VXLAN business traffic interruption

Symptom

A VXLAN network cannot forward Layer 3 VXLAN business traffic.

Common causes

The following are the common causes of this type of issue:

·     ACs or VXLAN tunnels are not established.

·     The device's router MAC address is incorrect.

Troubleshooting flow

Figure 4 shows the troubleshooting flowchart.

Figure 4 Flowchart for troubleshooting Layer 3 VXLAN business traffic interruption

 

Solution

To resolve the issue:

1.     Execute the display l2vpn vsi verbose command to view the VXLAN tunnels and ACs of the involved VSI.

<Sysname> display l2vpn vsi verbose

VSI Name: vpna

  VSI Index               : 0

  VSI State               : Up

  MTU                     : 1500

  Bandwidth               : Unlimited

  Broadcast Restrain      : Unlimited

  Multicast Restrain      : Unlimited

  Unknown Unicast Restrain: Unlimited

  MAC Learning            : Enabled

  MAC Table Limit         : -

  MAC Learning rate       : -

  Drop Unknown            : -

  Flooding                : Enabled

  Statistics              : Disabled

  VXLAN ID                : 10

  Tunnels:

    Tunnel Name          Link ID    State  Type      Flood proxy

    Tunnel1              0x5000001  Up     Manual    Disabled

  ACs:

    AC                               Link ID    State    Type

    GE0/0/1 srv1000                  0          Up       Manual

¡     If both the ACs and VXLAN tunnels are up, go to step 2.

¡     If an AC is down, modify the incorrect AC configuration.

¡     If a VXLAN tunnel is down, troubleshoot the issue as described in "Inter-VXLAN tunnel setup failure in an EVPN VXLAN network with distributed gateways."

2.     Execute the display evpn routing-table command, check the routing table of the L3VNI VPN instance, and record the nexthop address (Nexthop field) in the route for the target endpoint IP address (IP address field).

<Sysname> display evpn routing-table vpn-instance vpn1

Flags: E - with valid ESI   A – A-D ready   L - Local ES exists

 

VPN instance name: vpn1                             Local L3VNI: 7

IP address       Nexthop          Outgoing interface    NibID       Flags

10.1.1.11        1.1.1.1          Vsi-interface3        0x18000000  EAL

3.     Execute the display arp command to view the ARP information for the next hop.

<Sysname> display arp

  Type: S-Static   D-Dynamic   O-Openflow   R-Rule   M-Multiport  I-Invalid

IP address      MAC address    VLAN/VSI name Interface                Aging Type

1.1.1.1         00e0-fe50-6503 vsi1          Tunnel1                  960   D

¡     If the nexthop address is mapped to the router MAC address, go to step 4. Execute the display interface vsi-interface command to view the MAC address of the L3VNI VSI interface, which is also the router MAC address.

¡     If the nexthop address is not mapped to the router MAC address, restore consistency between the mapped MAC address and the MAC address of the L3VNI VSI interface. Alternatively, use the evpn global-mac command to configure an EVPN global MAC address.

4.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

¡     Diagnostic information collected by using the display diagnostic-information command.

Related alarm and log messages

Alarm messages

None.

Log messages

None.

Prolonged VM migration

Symptom

In an EVPN network, a VTEP does not learn the MAC address or ARP information of a VM immediately after the VM migrates to the VTEP.

Common causes

The following are the common causes of this type of issue:

·     The VTEP has not learned the MAC address or ARP entry for the migrated VM.

·     The VTEP has not learned the MAC address or ARP entry for the migrated VM through BGP EVPN route synchronization.

·     The BGP EVPN routes synchronized between VTEPs are not optimal ones.

Troubleshooting flow

Figure 5 shows the troubleshooting flowchart.

Figure 5 Flowchart for troubleshooting prolonged VM migration

 

Solution

1.     After the migration, check for the MAC address and ARP entry on the destination VTEP.

Execute the display l2vpn mac-address command and check the VSI MAC address table for the MAC address of the migrated VM.

<Sysname> display l2vpn mac-address

* - The output interface is issued to another VSI

MAC Address    State     VSI Name                        Link ID/Name   Aging

52f6-bc1e-0d06 EVPN      aaa                             Tunnel10       NotAging

0001-0001-0001 Dynamic   vpna                            GE0/0/1        Aging

--- 2 mac address(es) found  ---

Execute the display arp command and check the VSI ARP table for the ARP entry of the migrated VM.

<Sysname> display arp

  Type: S-Static   D-Dynamic   O-Openflow   R-Rule   M-Multiport  I-Invalid

IP address      MAC address    VLAN/VSI name Interface                Aging Type

10.1.1.3        0001-0001-0001 vpna          GE0/0/1                  960   D

1.1.1.4         00e0-fe60-5000 vsi2          Tunnel1                  --    M

¡     If a MAC address or ARP entry exists for the migrated VM, go to step 2.

¡     If no MAC address or ARP entry exists for the migrated VM, the VTEP does not learn the MAC address or ARP information of the migrated VM. Bring up the VM on the VTEP for the VTEP to learn the MAC address and ARP information.

2.     Before migration, check on the VTEP whether the MAC address or ARP of the migrated VM has been synchronized through BGP EVPN routes.

Execute the display evpn route mac command to verify that the MAC address of the migrated VM has been learned from synchronized BGP EVPN routes.  The value B in the Flags field indicates that a MAC address entry is learned from BGP EVPN routes.

<Sysname> display evpn route mac

Flags: D - Dynamic   B - BGP      L - Local active

       G - Gateway   S - Static   M - Mapping        I - Invalid

 

VSI name: bbb

EVPN instance: -

MAC address     Link ID/Name   Flags   Encap           Next hop

0000-0000-000a  1              DL      VXLAN           -

0001-0001-0001  Tunnel1        B       VXLAN           2.2.2.2

Execute the display evpn route arp command to verify that the ARP information of the migrated VM has been learned from synchronized BGP EVPN routes.  The value B in the Flags field indicates that an ARP entry is learned from BGP EVPN routes.

<Sysname> display evpn route arp

Flags: D - Dynamic   B - BGP      L - Local active

       G - Gateway   S - Static   M - Mapping        I - Invalid

 

VPN instance: vpn1                            Interface: Vsi-interface1

IP address      MAC address     Router MAC      VSI index   Flags

10.1.1.1        0001-0001-0001  a0ce-7e40-0400  0           B

10.1.1.11       0001-0001-0002  a0ce-7e40-0400  0           DL

10.1.1.101      0001-0011-0101  a0ce-7e40-0400  0           SL

10.1.1.102      0001-0011-0102  0011-9999-0000  0           BS

¡     If a MAC address entry or ARP entry has been learned through BGP EVPN route synchronization for the migrated VM, go to step 3.

¡     If no MAC address entry or ARP entry has been learned through BGP EVPN route synchronization for the migrated VM, execute the vpn-target command to modify the route targets of the local EVPN instance. Make sure the EVPN instance's route targets on the local and peer ends are consistent.

3.     Execute the display bgp l2vpn evpn command and verify that the MAC/IP advertisement route carrying the MAC address and ARP information of the migrated VM is optimal. Verify that the values in the State field include best. In the following output, the MAC/IP advertisement route advertises MAC address 0001-0203-0405 and IP address 5.5.5.5/32, and the route state values include best.

<Sysname> display bgp l2vpn evpn route-distinguisher 1.1.1.1:100 [2][5][48][0001-0203-0405][32][5.5.5.5] 136

 

 BGP local router ID: 172.16.250.133

 Local AS number: 100

 

 

 Route distinguisher: 1.1.1.1:100

 Total number of routes: 1

 Paths:   1 available, 1 best

 

 BGP routing table information of [2][5][48][0001-0203-0405][32][5.5.5.5]/136:

 From            : 10.1.1.2 (192.168.56.17)

 Rely nexthop    : 10.1.1.2

 Original nexthop: 10.1.1.2

 OutLabel        : NULL

 Ext-Community   : <RT: 1:2>, <RT: 1:3>, <RT: 1:4>, <RT: 1:5>, <RT: 1:6>, <RT: 1:7

                   >, <Encapsulation Type: VXLAN>, <Router's Mac: 0006-0708-0910

                   >, <MAC Mobility: Flag 0, SeqNum 2>, <Default GateWay>

 RxPathID        : 0x0

 TxPathID        : 0x0

 AS-path         : 200

 Origin          : igp

 Attribute value : MED 0, pref-val 0

 State           : valid, external, best

 IP precedence   : N/A

 QoS local ID    : N/A

 Traffic index   : N/A

 EVPN route type : MAC/IP advertisement route

 ESI             : 0001.0203.0405.0607.0809

 Ethernet tag ID : 5

 MAC address     : 0001-0001-0001

 IP address      : 10.1.1.1/32

 MPLS label1     : 10

 MPLS label2     : 100

 Re-origination  : Enable

¡     If the MAC/IP advertisement route is an optimal route, go to step 4.

¡     If the MAC/IP advertisement route is an optimal route, modify routing configuration for the route to be an optimal one.

4.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

¡     Diagnostic information collected by using the display diagnostic-information command.

Related alarm and log messages

Alarm messages

None.

Log messages

None.

Unavailability to access a VM after MAC migration

Symptom

After MAC address migration occurs on a VTEP, endpoints cannot access the migrated VM.

Common causes

The common causes of this issue are traffic anomalies and an incorrect outgoing interface in the MAC address entry for the VM due to network attacks.

Troubleshooting flow

Troubleshoot the issue by using the following process:

1.     View the MAC address migration information.

2.     Verify that the outgoing interface in the MAC address entry for the VM is correct.

Solution

1.     Execute the display evpn route mac-mobility command to view the MAC address migration information. In the following output, MAC address 1000-0000-0000 has migrated from GE0/0/1 to the local VTEP.

<Sysname> display evpn route mac-mobility

Flags: S - Suppressed, N - Not suppressed

  Suppression threshold: 5

  Detection cycle      : 180s

  Suppression time     : Permanent

 

VSI name      : vsia

EVPN instance : -

  MAC address     Move count Moved from               Flags Suppressed at

  1000-0000-0000  10         GE0/0/1                  S     15:30:30 2018/03/30

2.     Execute the display l2vpn mac-address command to verify that the outgoing interface in the MAC address entry for the VM is correct. The Link ID/Name field displays the name of the interface or tunnel interface where a MAC address is learned.

<Sysname> display l2vpn mac-address

* - The output interface is issued to another VSI

MAC Address    State     VSI Name                        Link ID/Name   Aging

1000-0000-0000 EVPN      aaa                             Tunnel10       NotAging

52f6-bc1e-0d06 Dynamic   vpna                            GE0/0/1        Aging

--- 2 mac address(es) found  ---

¡     If the outgoing interface is correct, go to step 3.

¡     If the outgoing interface is incorrect, bring up the VM on the destination VTEP for the VTEP to learn or update its forwarding entries.

3.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

¡     Diagnostic information collected by using the display diagnostic-information command.

Related alarm and log messages

Alarm messages

None.

Log messages

None.