Product series	Software version
SR8800-X	R8380PXX
SR8800-X-S	R8385PXX
SR8800-F	R8385PXX
CR16000-F	R8385PXX
CR16000-M	R8385PXX
vBRAS1000-CP	E2021P20 or higher
vBRAS1000-vUP	E3021P20 or higher

Prerequisites

This document provides generic BRAS services troubleshooting procedures for H3C BRAS devices. Some of the information might not apply to your device depending on its software and hardware version.

The interface numbers in this documentation are for illustration only. They might differ from the interface numbers available on your device.

The following information is provided based on the assumption that you have basic knowledge of BRAS services and are familiar with H3C BRAS devices.

General troubleshooting flow and diagnostic information collection for BRAS services

General troubleshooting flow

The following information provides a general high-level troubleshooting procedure for quick isolation of the problematic module and failure cause. You can modify this procedure based on your expertise and experience for effective troubleshooting of issues that differ in severity and complexity.

1. Identify the service impact scope of the failure.

Identify the following items:

¡ Affected subscriber services (for example, broadband and IPTV).

¡ The access services (for example, PPPoE and IPoE) used on the BRAS device to deliver the subscriber services.

¡ The number of affected users.

2. Identify the network topology.

This step is essential to troubleshooting BRAS issues, which are typically pertinent to the network.

3. Identify manual operations done on the network before and after the issue occurs.

Manual operations include configuration change and business cutover. This step helps narrow down the triggers of the issue quickly.

4. Analyze the characteristics of the affected users to find out if they have anything in common.

Examples of commonalities include the same access mode and the same Layer 2 switch.

5. Identify the point of failure.

Many times, network issues are caused by non-BRAS devices on the network. After you rule out the BRAS device, assist the customer in identifying the point of failure by using tools such as QoS flow statistics and port mirroring.

6. Identify the severity of the issue impact.

This step determines the action to take.

¡ If the impact is severe, quickly gather user information and take prompt action to restore services.

¡ If the impact is trivial, preferentially identify the cause of the issue and then remove the issue.

General BRAS troubleshooting procedures by plane

BRAS troubleshooting is divided into control plane troubleshooting and data plane troubleshooting.

· Control plane—Establishes, controls, and maintains network connectivity. It contains routing, signaling, and control protocols for routing, MPLS, and link layer connectivity. The protocols in the control plane generate and issue forwarding entries to the data plane to control its forwarding behaviors.

· Data plane—Also called the forwarding plane. It contains functionalities for receiving packets (including packets destined for the local node), forwarding data packets destined for remote nodes, and sending locally generated packets. Examples of data plane functionalities include the IPv4 and IPv6 protocol stacks, sockets, and functionalities that forward packets based on the forwarding tables at different layers.

General troubleshooting procedure for the control plane

Figure 1 shows the components used for BRAS user authentication and access. The User Connection Management (UCM) component is the bridge between the other components. It facilitates interaction between the components and assists in the establishment, maintenance, and termination of user connections.

Figure 1 Basic components used for BRAS user authentication and access

The following information describes the basic functionality of each component:

· User access identification component—Identifies and processes various user access protocol packets and obtains important user information such usernames, passwords, and physical locations during authentication. This information helps ensure secure and legitimate user access.

· UCM—Connects the other components to facilitate interaction between them and assists in the establishment, maintenance, and termination of user connections.

· AAA—Works with the AAA server to provide authentication, authorization, and accounting for users.

· Address management component—Allocates IP addresses to access users, and ensure proper use of IP resources through unified IP address management.

· Service control component—Controls the privileges, bandwidth, and QoS policies for the users to access basic services and value-added services.

The following information provides the general procedure to troubleshoot the control plane:

1. Collect information about the affected users, including their usernames, MAC addresses, and VLANs.

Execute the trace access-user command to trace the network access flow for an affected user, from login and authentication to address allocation. You can use the debugging output from this command to identify the phase in which the failure occurred.

[bras] trace access-user object 1 ?

access-mode Specify users by access mode

c-vlan Specify users by Customer-VLAN

calling-station-id Specify users by calling station ID

interface Specify users by interface

ip-address Specify a user by IP address

mac-address Specify users by MAC address

s-vlan Specify users by Service-VLAN

tunnel-id Specify users by tunnel ID

username Specify a user by username

2. Examine the configuration for the identified erroneous point and correct the misconfiguration, if any.

3. If the configuration is correct, examine the related modules such as the access, AAA (or RADIUS), address allocation, portal, and L2TP modules for errors.

NOTE:

After you specify a traced object by using the trace access-user command, you can use the display trace access-user command to view the configuration for the traced object. This command also displays the remaining amount of time for the trace session. When the remaining amount of time becomes 0, the trace session expires. To trace the same object, you must reconfigure it.

General troubleshooting procedure for the data plane

H3C BRAS devices provide hardware-based forwarding. The data plane is not error prone. If you receive reports on data traffic issues such as inaccurate rate limiting, packet loss, or loss of connectivity, take the following actions:

1. Verify that the user is online.

2. Verify that the rate limit and other authorization attributes assigned by the server to the user are correct.

3. Verify that data traffic from the user can arrive at the BRAS device.

4. If the issue persists, collect fault information and contact technical support for help.

Collecting user information

Service restoration is the top priority in dealing with a service outage while troubleshooting typically takes time. It is not always possible to promptly identify the cause of service outage solely based on debugging information. To assist in later troubleshooting, you must collect user information while restoring services.

The following are the best practices for user information collection:

· If only one user is affected, collect data that each module has for the affected user and some of the unaffected users to do a comparative analysis.

· If multiple users are affected, collect information about all affected users as soon as possible and contact technical support.

User information collection is to collect information about online users and users that were logged off abnormally. H3C BRAS devices offer a broad set of commands for you to collect user information. The following information describes only those used most commonly.

Support for the parameters in the commands described in this document differs depending on the hardware platform and software version.

Collecting information about online users

This task collects information about normal online users and temporary users, as well residual user information that should have been deleted.

Before you use the commands in this document to collect user information for troubleshooting purposes, read the command reference for the device to identify what information each parameter can produce. This will help you collect useful information efficiently.

For example, to collect complete information about a single user, execute the commands with the verbose keyword.

Collecting information for troubleshooting the PPPoE module

1. Execute the following command to collect information about PPP users that use the PPPoE access service. This command is the primary command you use to collect information about PPP users.

<Sysname> display access-user user-type pppoe ?

> Redirect it to a file

>> Redirect it to a file in append mode

auth-type Specify a user by authentication type

count Display the total number of users

domain Specify users by ISP domain

interface Specify users by interface

ip-pool Specify users by an IP pool

ip-pool-group Specify users by an IP pool group

ip-type Specify users by IP type

ipv6-address-protocol Specify users by IPv6 address protocol

ipv6-pool Specify users by an IPv6 pool

ipv6-pool-group Specify users by an IPv6 pool group

lac-ip Specify users by the IP address of an LAC

lns-ip Specify users by the IP address of an LNS

mac-address Specify a user by MAC address

remote-name Specify users by the tunnel name

slot Specify the slot number

start-time Specify users by the start time of coming online

user-address-type Specify users by address type

user-group Specify users by a user group

username Specify a user by username

verbose Display detailed information about users

vpn-instance Specify a VPN instance

vxlan Specify users by a range of VXLANs

| Matching output

<cr>

2. Execute the following command to collect statistics and information on the PPPoE server for online users.

<Sysname> display pppoe-server ?

chasten PPPoE connection blocking

packet Packet statistics

session PPPoE session information

throttled-mac Throttled MAC information

Collecting information for troubleshooting the IPoE module

1. Execute the following command to collect information about IPoE users, including IPoE Web users.

<Sysname> display access-user auth-type ?

admin Admin authentication

bind Bind authentication

dot1x 802.1X authentication

dvpn Dynamic VPN authentication

ike IKE authentication

mac-auth Mac authentication

portal Portal authentication

ppp PPP authentication

pre-auth Pre web authentication

sslvpn SSL VPN authentication

web-auth Web authentication

2. Execute the following command to collect information about IPoE bind authentication users.

<Sysname> display access-user auth-type bind ?

> Redirect it to a file

>> Redirect it to a file in append mode

count Display the total number of users

domain Specify users by ISP domain

interface Specify users by interface

ip-pool Specify users by an IP pool

ip-pool-group Specify users by an IP pool group

ip-type Specify users by IP type

ipv6-address-protocol Specify users by IPv6 address protocol

ipv6-pool Specify users by an IPv6 pool

ipv6-pool-group Specify users by an IPv6 pool group

lac-ip Specify users by the IP address of an LAC

lns-ip Specify users by the IP address of an LNS

mac-address Specify a user by MAC address

remote-name Specify users by the tunnel name

slot Specify the slot number

start-time Specify users by the start time of coming online

user-address-type Specify users by address type

user-group Specify users by a user group

user-type Specify users by type

username Specify a user by username

verbose Display detailed information about users

vpn-instance Specify a VPN instance

vxlan Specify users by a range of VXLANs

| Matching output

<cr>

Collecting information for troubleshooting the portal module

The device provides IPoE Web service in place of portal Web service. You collect information about IPoE Web service users through the commands for the IPoE module.

Collecting information for troubleshooting the L2TP module

1. Execute the following command to collect information about L2TP sessions.

<Sysname> display l2tp session ?

> Redirect it to a file

>> Redirect it to a file in append mode

lac Display L2TP session information of LAC

lns Display L2TP session information of LNS

local-address Specify sessions by the local IP address

remote-address Specify sessions by the remote IP address

statistics Statistics information

temporary L2TP temporary session information

tunnel-id Specify sessions by the specified local tunnel ID

username Specify sessions by the username

verbose Display detailed L2TP session information

| Matching output

<cr>

2. Execute the following command to collect information about temporary L2TP sessions.

<Sysname> display l2tp session temporary ?

> Redirect it to a file

>> Redirect it to a file in append mode

| Matching output

<cr>

3. Execute the following command to collect information about L2TP tunnels.

<Sysname> display l2tp tunnel ?

> Redirect it to a file

>> Redirect it to a file in append mode

group-name Specify tunnels by the group name

group-number Specify tunnels by the group number

lac Display L2TP tunnel information of LAC

lns Display L2TP tunnel information of LNS

local-address Specify tunnels by the local IP address

remote-address Specify tunnels by the remote IP address

statistics Statistics information

tunnel-id Specify tunnels by the local L2TP tunnel ID

tunnel-name Specify tunnels by the remote tunnel name

verbose Display detailed L2TP tunnel information

vsrp L2TP VSRP tunnel information

| Matching output

<cr>

4. Execute the following command on the LAC to collect information about PPP users that access the network through L2TP.

<Sysname> display access-user user-type lac ?

> Redirect it to a file

>> Redirect it to a file in append mode

auth-type Specify a user by authentication type

count Display the total number of users

domain Specify users by ISP domain

interface Specify users by interface

ip-pool Specify users by an IP pool

ip-pool-group Specify users by an IP pool group

ip-type Specify users by IP type

ipv6-address-protocol Specify users by IPv6 address protocol

ipv6-pool Specify users by an IPv6 pool

ipv6-pool-group Specify users by an IPv6 pool group

lac-ip Specify users by the IP address of an LAC

lns-ip Specify users by the IP address of an LNS

mac-address Specify a user by MAC address

remote-name Specify users by the tunnel name

slot Specify the slot number

start-time Specify users by the start time of coming online

user-address-type Specify users by address type

user-group Specify users by a user group

username Specify a user by username

verbose Display detailed information about users

vpn-instance Specify a VPN instance

vxlan Specify users by a range of VXLANs

| Matching output

<cr>

5. Execute the following command on the LNS to collect information about PPP users that access the network through L2TP.

<Sysname> display access-user user-type lns ?

> Redirect it to a file

>> Redirect it to a file in append mode

auth-type Specify a user by authentication type

count Display the total number of users

domain Specify users by ISP domain

interface Specify users by interface

ip-pool Specify users by an IP pool

ip-pool-group Specify users by an IP pool group

ip-type Specify users by IP type

ipv6-address-protocol Specify users by IPv6 address protocol

ipv6-pool Specify users by an IPv6 pool

ipv6-pool-group Specify users by an IPv6 pool group

lac-ip Specify users by the IP address of an LAC

lns-ip Specify users by the IP address of an LNS

mac-address Specify a user by MAC address

remote-name Specify users by the tunnel name

slot Specify the slot number

start-time Specify users by the start time of coming online

user-address-type Specify users by address type

user-group Specify users by a user group

username Specify a user by username

verbose Display detailed information about users

vpn-instance Specify a VPN instance

vxlan Specify users by a range of VXLANs

| Matching output

<cr>

Collecting information for troubleshooting the DHCP module

1. Collect information about the idle IP addresses available for allocation on the DHCP server.

<Sysname> display dhcp server free-ip ?

> Redirect it to a file

>> Redirect it to a file in append mode

pool Specify a DHCP pool

vpn-instance Specify a VPN instance

| Matching output

<cr>

2. Collect information about the allocated IP addresses that are in use on the DHCP server.

<Sysname> display dhcp server ip-in-use ?

> Redirect it to a file

>> Redirect it to a file in append mode

interface Specify the interface

ip Specify an IP address

pool Specify a DHCP pool

subnet Specify s subnet

up-backup-group Specify a UPBACKUPGROUP

up-id Specify a UP Id

vpn-instance Specify a VPN instance

vxlan Specify a VXLAN

| Matching output

<cr>

3. Collect information about IP and MAC bindings in expired leases on the DHCP server.

<Sysname> display dhcp server expired ?

> Redirect it to a file

>> Redirect it to a file in append mode

interface Specify the interface

ip Specify an IP address

mac Specify a MAC address

pool Specify a DHCP pool

up-backup-group Specify a UPBACKUPGROUP

up-id Specify a UP Id

verbose Detailed information

vpn-instance Specify a VPN instance

vxlan Specify a VXLAN

| Matching output

<cr>

4. Collect information about IP and MAC bindings recorded for IP address conflict on the DHCP server.

<Sysname> display dhcp server conflict ?

> Redirect it to a file

>> Redirect it to a file in append mode

interface Specify the interface

ip Specify an IP address

up-backup-group Specify a UPBACKUPGROUP

up-id Specify a UP Id

vpn-instance Specify a VPN instance

vxlan Specify a VXLAN

| Matching output

<cr>

5. Collect information about client address entries recorded on the DHCP relay agent.

<Sysname> display dhcp relay client-information ?

> Redirect it to a file

>> Redirect it to a file in append mode

interface Specify the interface

ip Specify an IP address

| Matching output

<cr>

Collecting information for troubleshooting the AAA module

No commands are available for the AAA module to record user information. To obtain information about AAA users, use the information recorded by the access modules.

Collecting information about abnormally logged-off users

You collect information about abnormally logged-off users for analysis of the recorded logoff reasons and message exchanges between modules to identify the root cause of the abnormal logoffs.

Collecting information for troubleshooting the PPPoE module

1. Collect PPPoE server negotiation packet statistics.

<Sysname> display pppoe-server packet statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

slot Specify the slot number

| Matching output

<cr>

2. Collect PPP negotiation packet statistics.

<Sysname> display ppp packet statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

slot Specify the slot number

| Matching output

<cr>

3. Collect the offline records for login users.

<Sysname> display aaa offline-record access-type ppp ?

> Redirect it to a file

>> Redirect it to a file in append mode

brief Display brief information

count Specify the number of records to be displayed

domain Specify an ISP domain

interface Specify an interface

ip Specify an IPv4 address

ipv6 Specify an IPv6 address

mac-address Specify a MAC address

s-vlan Specify a service provider network VLAN

slot Specify the slot number

username Specify a username

| Matching output

<cr>

Collecting information for troubleshooting the IPoE module

1. Collect information about abnormally logged-off DHCP clients.

<Sysname> display ip subscriber abnormal-logout ?

> Redirect it to a file

>> Redirect it to a file in append mode

interface Specify an interface

ip Specify the IP address

ip-type Specify users by IP type

ipv6 Specify the IPv6 address

mac Specify a MAC address

slot Specify the slot number

verbose Detailed information

| Matching output

<cr>

2. Collect the offline records for IPoE users.

<Sysname> display aaa offline-record access-type ipoe ?

> Redirect it to a file

>> Redirect it to a file in append mode

brief Display brief information

count Specify the number of records to be displayed

domain Specify an ISP domain

interface Specify an interface

ip Specify an IPv4 address

ipv6 Specify an IPv6 address

mac-address Specify a MAC address

s-vlan Specify a service provider network VLAN

slot Specify the slot number

username Specify a username

| Matching output

<cr>

3. Collect statistics for IPoE users.

<Sysname> display access-user count ?

> Redirect it to a file

>> Redirect it to a file in append mode

| Matching output

<cr>

Collecting information for troubleshooting the L2TP module

1. Collect L2TP protocol packet statistics.

<Sysname> display l2tp control-packet statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

summary Summary L2TP control packet statistics

tunnel L2TP control packet statistics of each tunnel

| Matching output

<cr>

2. Collect L2TP statistics.

<Sysname> display l2tp statistics ?

all All L2TP statistics

rdbm RedisDBM statistics

vsrp VSRP statistics

Collecting information for troubleshooting the DHCP module

1. Collect DHCP server statistics.

<Sysname> display dhcp server statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

pool Specify a DHCP pool

vpn-instance Specify a VPN instance

| Matching output

<cr>

2. Collect DHCP relay statistics.

<Sysname> display dhcp relay packet statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

interface Specify the interface

| Matching output

<cr>

Collecting information for troubleshooting the AAA module

1. Collect the abnormal offline records maintained by the AAA module.

<Sysname> display aaa abnormal-offline-record ?

> Redirect it to a file

>> Redirect it to a file in append mode

access-type Specify an access type

domain Specify an ISP domain

interface Specify an interface

ip Specify an IPv4 address

ipv6 Specify an IPv6 address

mac-address Specify a MAC address

offline-reason Specify a user offline reason

s-vlan Specify a service provider network VLAN

slot Specify the slot number

time Specify a time range

username Specify a username

| Matching output

<cr>

2. Collect the normal offline records maintained by the AAA module.

<Sysname> display aaa normal-offline-record ?

> Redirect it to a file

>> Redirect it to a file in append mode

access-type Specify an access type

domain Specify an ISP domain

interface Specify an interface

ip Specify an IPv4 address

ipv6 Specify an IPv6 address

mac-address Specify a MAC address

s-vlan Specify a service provider network VLAN

slot Specify the slot number

time Specify a time range

username Specify a username

| Matching output

<cr>

3. Collect the offline records maintained by the AAA module.

<Sysname> display aaa offline-record ?

> Redirect it to a file

>> Redirect it to a file in append mode

access-type Specify an access type

domain Specify an ISP domain

interface Specify an interface

ip Specify an IPv4 address

ipv6 Specify an IPv6 address

mac-address Specify a MAC address

s-vlan Specify a service provider network VLAN

slot Specify the slot number

time Specify a time range

username Specify a username

| Matching output

<cr>

4. Collect the user online failure records maintained by the AAA module.

<Sysname> display aaa online-fail-record ?

> Redirect it to a file

>> Redirect it to a file in append mode

access-type Specify an access type

domain Specify an ISP domain

interface Specify an interface

ip Specify an IPv4 address

ipv6 Specify an IPv6 address

mac-address Specify a MAC address

s-vlan Specify a service provider network VLAN

slot Specify the slot number

time Specify a time range

username Specify a username

| Matching output

<cr>

5. Collect the RADIUS packet statistics maintained by the AAA module.

<Sysname> display radius statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

server Specify a RADIUS server

| Matching output

<cr>

6. Collect load statistics for all RADIUS servers.

<Sysname> display radius server-load statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

| Matching output

<cr>

7. Collect the statistics maintained by the RADIUS module for the online access users in ISP domains.

<Sysname> display domain access-user statistics ?

> Redirect it to a file

>> Redirect it to a file in append mode

| Matching output

<cr>

BRAS service troubleshooting procedures at a glance

Troubleshooting procedures for campus networks

The troubleshooting procedures listed in Table 2 apply to the following router series:

· SR8800-X.

· SR8800-X-S.

· SR8800-F.

· CR16000-F.

· CR16000-M.

Support for the listed procedures differs depending on the router series.

Use Table 2 to quickly locate the troubleshooting procedure of interest by failure type.

Table 2 BRAS service troubleshooting procedures for campus networks

Failure type	Troubleshooting procedures
Troubleshooting user online failures and abnormal offline events	· PPPoE user online failures and abnormal offline events · PPPoE agency user online failures and abnormal offline events · Campus user failures to access the external network on a PPPoE agency network · L2TP user online failures and abnormal offline events · IPoE: ¡ IPoE user online failures and abnormal offline events ¡ IPoE DHCP user online failures and abnormal offline events ¡ IPoE NDRS user online failures and abnormal offline events ¡ IPoE static user online failure or abnormal offline event ¡ IPoE Web user online failure

Failure type

Troubleshooting procedures

Troubleshooting user online failures and abnormal offline events

· PPPoE user online failures and abnormal offline events

· PPPoE agency user online failures and abnormal offline events

· Campus user failures to access the external network on a PPPoE agency network

· L2TP user online failures and abnormal offline events

· IPoE:

¡ IPoE user online failures and abnormal offline events

¡ IPoE DHCP user online failures and abnormal offline events

¡ IPoE NDRS user online failures and abnormal offline events

¡ IPoE static user online failure or abnormal offline event

¡ IPoE Web user online failure

Troubleshooting procedures for carrier networks

Table 3 lists the troubleshooting procedures for the following router series:

· CR16000-F.

· SR8800-F.

· vBRAS1000-CP.

· vBRAS1000-vUP.

Support for the listed procedures differs depending on the router model.

Control-/user plane separation (CUPS) networks use the same troubleshooting procedures as non-CUPS networks. This document uses a non-CUPS network for example to describe the troubleshooting procedures.

IMPORTANT:

· Before you use this guide to troubleshoot BRAS services on a CUPS network, make sure you are familiar with the CUPS network architecture and the configuration for service modules, especially the configuration specific to service modules such as PPPoE and L2TP. This will help you troubleshoot BRAS issues more quickly.

· On a CUPS network, execute the commands in this document on the control plane (CP) devices unless otherwise stated.

For information about the CUPS network architecture, see CP and UP separation basics in the CP and UP separation configuration guide for the BRAS device. For information about configuring a service module, see the configuration guide that come with the BRAS device for that module.

Use Table 3 to quickly locate the troubleshooting procedure of interest by failure type on a telecom network.

Table 3 BRAS service troubleshooting procedures for carrier networks

Failure type

Troubleshooting procedures

Troubleshooting user online failures and abnormal offline events

· PPPoE user online failures and abnormal offline events

· PPPoE agency user online failures and abnormal offline events

· Campus user failures to access the external network on a PPPoE agency network

· L2TP user online failures and abnormal offline events

· IPoE issues:

¡ IPoE user online failures and abnormal offline events

¡ IPoE DHCP user online failures and abnormal offline events

¡ IPoE NDRS user online failures and abnormal offline events

¡ IPoE static user online failure or abnormal offline event

¡ IPoE Web user online failure

Troubleshooting issues specific to a CUPS network

· User online failure

· CP-UP connection management issues:

¡ CUPS channel failure

¡ Management channel establishment failure

¡ Packet forwarding failure for the management channel

¡ Control channel establishment failure

¡ Packet forwarding failure for the control channel

¡ Protocol channel establishment failure

¡ Packet forwarding failure for the protocol channel

· Auto scaling issues:

¡ VM manual scaling failure

¡ VM auto scaling failure

¡ CP disaster recovery

· UP backup issues:

¡ Master/backup interface failure or master/backup switchover

¡ Long master/backup interface switchover

¡ Two master interfaces on UPs

¡ Two backup interfaces on UPs

· VM issues:

¡ Image file upload failure

¡ VNF package upload failure

¡ VM deployment failure

¡ VM creation or startup failure due to insufficient resources

¡ VM startup failure due to version file issues

¡ VM registration failure

¡ Subnet request and release failure of BRAS-VMs

¡ High CPU control core usage on a VM

¡ Memory alarm threshold crossings caused by high memory usages of VMs on a vBRAS-CP

· Attack protection issues

Troubleshooting user online failures and abnormal offline events

PPPoE user online failures and abnormal offline events

Symptom

A PPPoE user fails to come online or abnormally goes offline.

Common causes

The following are the common causes of this type of issue:

· A user enters an incorrect username or password.

· The number of consecutive authentication failures of a user exceeds the maximum number allowed, and the user is blocked. The blocking period has not expired.

· The configuration is incorrect. For example, no IP address pool is configured, or the IP addresses in the configured IP address pool are exhausted. As a result, a user cannot obtain an IP address.

· A user owes fees.

Troubleshooting flow

Figure 2 shows the troubleshooting flowchart.

Figure 2 Flowchart for troubleshooting PPPoE user online failures and abnormal offline events

Solution

1. View the PPPoE user online failure reasons.

Execute the display aaa online-fail-record command to display user online failure reasons.

<Sysname> display aaa online-fail-record username aaa

Username: aaa

Domain: test

MAC address: 0010-9400-0007

Access type: PPPoE

Access interface: Ten-GigabitEthernet3/1/1

SVLAN/CVLAN: -/-

IP address: -

IPv6 address: -

Online request time: 2019/09/23 14:57:06

Online failure reason: PPP negotiation terminated.

The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting. Search for the displayed reason in “Appendix A Reasons for user login failures and abnormal logouts” and troubleshoot according to the corresponding solution.

You can resolve the issues caused by some failure reasons (for example, Authentication method error or Local authentication request was rejected) by checking the configuration. If you cannot see the failure records for some failure reasons, proceed with the next step.

2. View the PPPoE user offline reasons.

If you cannot obtain the online failure reasons for a user in step 1, the user might come online successfully and then go offline. In this case, use the display aaa offline-record command to display user offline records.

<Sysname> display aaa offline-record

Total count: 1

Username: jay

Domain: dm1

MAC address: -

Access type: Telnet

Access interface: GigabitEthernet1/0/1

SVLAN/CVLAN: -/-

IP address: 19.19.0.2

IPv6 address: -

Online request time: 2020-01-02 15:20:33

Offline time: 2020-2-28 15:20:56

Offline reason: User request

If a user first comes online successfully and then goes offline, the Offline reason field in the command output displays the offline reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting.

Search for the displayed reason in “Appendix A Reasons for user login failures and abnormal logouts” and troubleshoot according to the corresponding solution.

If you cannot use the display aaa offline-record command to obtain the user offline reasons, proceed with the next step.

3. Verify that the PPPoE user settings are correct.

Troubleshoot the settings according to the manuals for BRASs. For example, see tasks at a glance or configuration examples in the corresponding manuals.

¡ If configuration errors exist, correct the configuration and then try to come online again.

¡ If the configuration is correct but the issue persists, proceed with the next step.

4. Identify whether the user is blocked by PPP.

Execute the display ppp chasten user command to identify whether the user is blocked by PPP.

¡ If the user is blocked, redial after the remaining blocking time expires according to the command output.

¡ If the user is not blocked, proceed with the next step.

5. Enable the service tracing messages.

Execute the trace access-user command to enable the service tracing feature for users to test user online events. After the user online process is completed, view the service tracing messages. If the device does not receive PADI or PADR packets, identify whether the Layer 2 network is reachable, the port state is normal, the access type is Layer 2, the authentication method contains PPP, and the interface is bound to a virtual-template interface.

6. Identify whether the user is blocked by PPPoE.

Execute the display pppoe-server chasten user command to identify whether the user is blocked by PPPoE.

¡ If the user is blocked, redial after the remaining blocking time expires according to the command output.

¡ If the user is not blocked, proceed with the next step.

7. Check the device failures.

If you cannot view any service tracing message for the user, check the following configurations:

¡ Make sure the physical connections of the device are correct.

¡ Make sure the configuration on the device is correct.

¡ Make sure the Layer 2 network configuration is correct.

¡ Make sure packets can reach the device.

In probe view, execute the display hardware internal rxtx packet statistic command to view statistics of packets received/sent by the device driver. Identify whether the user packets are sent to the BRAS. (non-vBRAS-CPs.)

On a CUPS network, identify whether the user packets are sent to the BRAS. For more information, see “User online failure.”

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal rxtx packet statistic slot 3 cpu 0

Net port packet loss count:

code counter

Rx packets statistic:

counter success rate

NET ->RXTX : 171883335 171554546 342 pps

Cpu code input list:(Mgment to L1 queue)

code counter success(whitelist/normal)

5 14475 14475(0/14475)

6 2308 2308(0/2308)

17 262 262(0/262)

26 1013133 986703(0/986703)

30 6014064 6014064(0/6014064)

35 282 282(0/282)

37 79280 79280(0/79280)

43 2423 2423(0/2423)

44 44438 44438(0/44438)

45 1181 1181(0/1181)

49 60638 60638(0/60638)

50 25 25(0/25)

51 60361 60361(0/60361)

52 496 496(0/496)

53 115767 115767(115726/41)

54 83228 83228(83228/0)

61 191235 191235(0/191235)

77 12007 11988(0/11988)

99 6041569 6041569(0/6041569)

106 30 30(0/30)

149 158129148 157826808(0/157826808)

175 16985 16985(16979/6)

Callback function packets statistic:

total(r) success(r) total(c) success(c)

MACL: 0 0 0 0

NATL: 0 0 0 0

BFD: 0 0 0 0

(null): 0 0 0 0

Task input pkt statistics:

Task name total success

Main Task : 165540452 165540452

Icmp Task : 30 30

Cpu code input list:(L2 queue to platform)

code counter success drop rate

5 14475 14475 0 0

6 2308 2308 0 0

17 262 262 0 0

26 986703 986703 0 1

35 282 282 0 0

37 79280 79280 0 0

43 2423 2423 0 0

44 44438 44438 0 0

45 1181 1181 0 0

49 60638 60638 0 0

50 25 25 0 0

51 60361 60361 0 0

52 496 496 0 0

53 115767 115767 0 0

54 83228 83228 0 0

61 191235 191235 0 0

77 11988 11988 0 0

99 6041569 6041569 0 12

106 30 30 0 0

149 157826808 157826808 0 314

175 16985 16985 0 0

Cpu code to protocol:

5 ARP_REQ_LOCAL

6 ARP_REL

17 ARP_REQ

26 PPPOE

30 DIAG

35 ND_NA

37 LLDP,CDP

43 ND_NS

44 ND_RS

45 ND_RA

49 OSPF_HELLO,OSPF_LSU,OSPF_LSACK

50 OSPF_DD,OSPF_LSR

51 OSPFV3_HELLO,OSPFV3_LSU,OSPFV3_LSACK

52 OSPFV3_DD,OSPFV3_LSR

53 LDP_HELLO

54 LDP_NOTIF,LDP_INIT,LDP_KPALV,LDP_ADDR,LDP_LABEL

61 DHCP_IPOE,DHCP_SNOOPING,DHCP,DHCPv6_RELAY,DHCPv6_RELS,DHCPv6_SERV

77 IP_SUBNET

99 PPPOE_PPP

106 ICMP,ICMPV6

149 L2TP

175 APP_TELNET

Debug packets statistic:

counter counter rate

NET->RXTX->SERVICE: 0 0 0 pps

SERVICE->RXTX->NET: 0 0 0 pps

failed

MbufTrSend: 0

FoundIfindex: 0

SaveCoreSta: 0

MainCoreSta: 0

TxFailedSta: 0

The 26 and 99 fields represent PPPoE and PPPoE_PPPP, respectively. If the received packet counts for 26 and 99 increase, it means that the device has received PPP/PPPoE packets and sent them to the platform. You can use debugging for the forwarding function to check the layer on which packets are dropped step by step. If the counts do not increase, execute the display hardware internal np pktcnt drop command to identify whether the driver has dropped packet count.

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal np pktcnt drop slot 3 (the command for viewing the packet count varies by device model)

Current Mcode Type: SIRIUS_RELEASE

The NP 0 is Both NP

Drop packet statistics

32B7 116497 TOPparse total discarded pkts

350F 916677 TOPresolve total discarded pkts

51A 66 PRS Ingress route interface deny L2 forward

56B 384 PRS Ingress Route interface deny L2 forward

63C 403633 RSV Ingress ARP packet FTN or BROADCAST table no ma

tch

63E 372789 RSV Ingress PROTOCOL_MAC and BROADCAST table no mat

641 161878 RSV Ingress PROTOCOL_MAC.THB is set, but BROADCAST

table no match

645 149489 RSV Ingress multicast, MULTICAST.DROP is set

646 144150 RSV Ingress multicast, match MULTICAST default entr

y, but BROADCAST table no match

663 4 RSV Ingress broadcast packets from route port, PROT

OCOL_PORT table no match

- If the dropped packet count keeps increasing, analyze the possible issues according to the packet drop reasons.

- If the dropped packet count does not increase and the number of packets sent to the CPU also does not increase, it means that packets are not successfully sent to the BRAS. In this case, collect the failure information and contact Technical Support.

Only if the preceding configurations are all correct, you can use the service tracing function to see the tracing messages.

If you determine that the user online failure reason is incorrect configuration, check the local configuration according to the tracing messages.

¡ For a RADIUS authentication user, you must identify whether the RADIUS server is correctly configured and the RADIUS server state is normal.

¡ For a local authentication user, identify whether the local account configuration is correct, and the number of access users is not limited.

8. Identify whether the LCP negotiation succeeds.

You can obtain the negotiation packet statistics on the BRAS and client separately (on the client, you can capture the negotiation packets). In this way, you can quickly locate what causes the LCP negotiation failure: the device, the client, or the cooperation between devices.

<Sysname> display ppp packet statistics

PPP packet statistics in slot 97:

-----------------------------------LCP--------------------------------------

SEND_LCP_CON_REQ : 6185 RECV_LCP_CON_REQ : 6177

SEND_LCP_CON_NAK : 0 RECV_LCP_CON_NAK : 0

SEND_LCP_CON_REJ : 0 RECV_LCP_CON_REJ : 0

SEND_LCP_CON_ACK : 6177 RECV_LCP_CON_ACK : 6000

SEND_LCP_CODE_REJ : 0 RECV_LCP_CODE_REJ : 0

SEND_LCP_PROT_REJ : 0 RECV_LCP_PROT_REJ : 0

SEND_LCP_TERM_REQ : 0 RECV_LCP_TERM_REQ : 0

SEND_LCP_TERM_ACK : 0 RECV_LCP_TERM_ACK : 0

SEND_LCP_ECHO_REQ : 0 RECV_LCP_ECHO_REQ : 0

SEND_LCP_ECHO_REP : 0 RECV_LCP_ECHO_REP : 0

SEND_LCP_FAIL : 0 SEND_LCP_CON_REQ_RETRAN : 185

-----------------------------------IPCP-------------------------------------

SEND_IPCP_CON_REQ : 0 RECV_IPCP_CON_REQ : 0

SEND_IPCP_CON_NAK : 0 RECV_IPCP_CON_NAK : 0

SEND_IPCP_CON_REJ : 0 RECV_IPCP_CON_REJ : 0

SEND_IPCP_CON_ACK : 0 RECV_IPCP_CON_ACK : 0

SEND_IPCP_CODE_REJ : 0 RECV_IPCP_CODE_REJ : 0

SEND_IPCP_PROT_REJ : 0 RECV_IPCP_PROT_REJ : 0

SEND_IPCP_TERM_REQ : 0 RECV_IPCP_TERM_REQ : 0

SEND_IPCP_TERM_ACK : 0 RECV_IPCP_TERM_ACK : 0

SEND_IPCP_FAIL : 0

-----------------------------------IPV6CP-----------------------------------

SEND_IPV6CP_CON_REQ : 0 RECV_IPV6CP_CON_REQ : 0

SEND_IPV6CP_CON_NAK : 0 RECV_IPV6CP_CON_NAK : 0

SEND_IPV6CP_CON_REJ : 0 RECV_IPV6CP_CON_REJ : 0

SEND_IPV6CP_CON_ACK : 0 RECV_IPV6CP_CON_ACK : 0

SEND_IPV6CP_CODE_REJ : 0 RECV_IPV6CP_CODE_REJ : 0

SEND_IPV6CP_PROT_REJ : 0 RECV_IPV6CP_PROT_REJ : 0

SEND_IPV6CP_TERM_REQ : 0 RECV_IPV6CP_TERM_REQ : 0

SEND_IPV6CP_TERM_ACK : 0 RECV_IPV6CP_TERM_ACK : 0

SEND_IPV6CP_FAIL : 0

-----------------------------------OSICP------------------------------------

SEND_OSICP_CON_REQ : 0 RECV_OSICP_CON_REQ : 0

SEND_OSICP_CON_NAK : 0 RECV_OSICP_CON_NAK : 0

SEND_OSICP_CON_REJ : 0 RECV_OSICP_CON_REJ : 0

SEND_OSICP_CON_ACK : 0 RECV_OSICP_CON_ACK : 0

SEND_OSICP_CODE_REJ : 0 RECV_OSICP_CODE_REJ : 0

SEND_OSICP_PROT_REJ : 0 RECV_OSICP_PROT_REJ : 0

SEND_OSICP_TERM_REQ : 0 RECV_OSICP_TERM_REQ : 0

SEND_OSICP_TERM_ACK : 0 RECV_OSICP_TERM_ACK : 0

SEND_OSICP_FAIL : 0

-----------------------------------MPLSCP-----------------------------------

SEND_MPLSCP_CON_REQ : 0 RECV_MPLSCP_CON_REQ : 0

SEND_MPLSCP_CON_NAK : 0 RECV_MPLSCP_CON_NAK : 0

SEND_MPLSCP_CON_REJ : 0 RECV_MPLSCP_CON_REJ : 0

SEND_MPLSCP_CON_ACK : 0 RECV_MPLSCP_CON_ACK : 0

SEND_MPLSCP_CODE_REJ : 0 RECV_MPLSCP_CODE_REJ : 0

SEND_MPLSCP_PROT_REJ : 0 RECV_MPLSCP_PROT_REJ : 0

SEND_MPLSCP_TERM_REQ : 0 RECV_MPLSCP_TERM_REQ : 0

SEND_MPLSCP_TERM_ACK : 0 RECV_MPLSCP_TERM_ACK : 0

SEND_MPLSCP_FAIL : 0

-----------------------------------AUTH-------------------------------------

SEND_PAP_AUTH_REQ : 0 RECV_PAP_AUTH_REQ : 6000

SEND_PAP_AUTH_ACK : 0 RECV_PAP_AUTH_ACK : 0

SEND_PAP_AUTH_NAK : 0 RECV_PAP_AUTH_NAK : 0

SEND_CHAP_AUTH_CHALLENGE: 0 RECV_CHAP_AUTH_CHALLENGE: 0

SEND_CHAP_AUTH_RESPONSE : 0 RECV_CHAP_AUTH_RESPONSE : 0

SEND_CHAP_AUTH_ACK : 0 RECV_CHAP_AUTH_ACK : 0

SEND_CHAP_AUTH_NAK : 0 RECV_CHAP_AUTH_NAK : 0

SEND_PAP_AUTH_FAIL : 0 SEND_CHAP_AUTH_FAIL : 0

Common symptoms include:

¡ During the LCP negotiation process of a PPPoE client, the PPPoE client sends config-requests, and the device responds and sends config-nak/config-reject packets. In this case, the client must modify the attribute values in the corresponding config-requests according to the replies from the device. However, the client might always not modify the negotiation attributes. As a result, the negotiation fails. In this case, you can capture packets or execute the debugging ppp all command to enable debugging to check the attributes that cause the negotiation failure. According to these attributes, you can check the corresponding configuration and make sure the configuration is correct. If the issue persists, contact Technical Support.

¡ The device is configured with CHAP authentication. However, the client supports only PAP authentication. Therefore, LCP negotiation always fails. In this case, modify CHAP authentication to PAP authentication on the device.

9. Identify whether authentication succeeds.

¡ For local authentication, the authentication failure reason might be:

- The local account does not exist.

- The authentication domain is not activated.

- The account is not activated.

- The account type is inconsistent.

- The access is limited.

¡ For RADIUS authentication, the authentication failure reason might be the device does not receive RADIUS replies or RADIUS authentication is rejected.

10. Identify whether the NCP negotiation succeeds.

Typically, NCP performs only address negotiation in PPPoE. Therefore, NCP negotiation failure means address negotiation failure. You can check the configuration according to the locally allocated address, RADIUS allocated address, and DHCP allocated address.

11. Identify whether accounting is normal.

If the user still cannot come online in this case, accounting might fail. The most common reason is that accounting fails to start. In this case, you must identify whether the device and AAA server can reach each other at Layer 3 and whether the AAA server’s accounting function is configured correctly.

12. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

PPPoE agency user online failures and abnormal offline events

Symptom

A PPPoE agency user fails to come online or abnormally goes offline.

Common causes

The following are the common causes of this type of issue:

· The campus BRAS user corresponding to a PPPoE agency user fails to come online or abnormally goes offline.

· The PPPoE agency configuration is incorrect. For example:

¡ The interface connecting the campus BRAS to the service provider BRAS is not enabled with PPPoE agency. As a result, a PPPoE agency user fails to come online.

¡ The PPPoE agency group name configured for the PPPoE agency interface on the campus BRAS is different from the PPPoE agency group name deployed through COA messages by the campus AAA server. As a result, a PPPoE agency user fails to come online.

¡ The undo pppoe-agency forward command is executed in user group view of a campus BRAS user to delete the PPPoE agency forwarding policy. As a result, the corresponding PPPoE agency user goes offline.

¡ The COA messages are used on the campus AAA server to modify the user-group attribute of a campus BRAS user to a user group that does not support PPPoE agency, or the undo user-group command is executed in system view on the campus BRAS to delete the user group of a campus BRAS user. As a result, the corresponding PPPoE agency user goes offline.

· The link between the campus BRAS and the service provider BRAS fails. For example, the PPPoE agency interface is down.

· The campus AAA server forcibly logs out a PPPoE agency user.

· A PPPoE agency user is forcibly logged out by the service provider because the user traffic is exhausted or the user owes fees.

Troubleshooting flow

Figure 3 shows the troubleshooting flowchart.

Figure 3 Flowchart for troubleshooting PPPoE agency user online failures and abnormal offline events

Solution

1. Identify whether the campus BRAS user corresponding to a PPPoE agency user has come online successfully.

Execute the display access-user command in any view on the campus BRAS to identify whether the campus BRAS user corresponding to a PPPoE agency user has come online successfully.

¡ If the campus BRAS user fails to come online or abnormally goes offline after coming online, resolve the issue according to the access authentication method (IPoE or PPPoE) used by the campus BRAS user and the online failure and abnormal offline failure troubleshooting flow for the user type in “Troubleshooting user online failures and abnormal offline events.”

¡ If the campus BRAS user comes online normally, proceed with the next step.

2. View the PPPoE agency user online failure reasons.

Execute the display aaa online-fail-record command in any view on the campus BRAS to identify the PPPoE agency user online failure reasons.

<Sysname> display aaa online-fail-record username aaa

Username: aaa

Domain: test

MAC address: 0010-9400-0007

Access type: PPPoEA

Access interface: Ten-GigabitEthernet3/1/1

SVLAN/CVLAN: -/-

IP address: -

IPv6 address: -

Online request time: 2022/04/23 14:57:06

Online failure reason: Disabled PPPoE agency.

The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason. Search for the displayed reason in “Appendix A Reasons for user login failures and abnormal logouts” and troubleshoot according to the corresponding solution.

If you cannot see the failure records for some failure reasons, proceed with the next step.

3. View the PPPoE agency user offline reasons.

If you cannot obtain the online failure reasons for a user in the display aaa online-fail-record command output, the user might come online successfully and then go offline. In this case, use the display aaa offline-record command to display user offline records.

<Sysname> display aaa offline-record

Total count: 1

Username: jay

Domain: dm1

MAC address: -

Access type: Telnet

Access interface: GigabitEthernet1/0/1

SVLAN/CVLAN: -/-

IP address: 19.19.0.2

IPv6 address: -

Online request time: 2020-01-02 15:20:33

Offline time: 2020-2-28 15:20:56

Offline reason: User request

Search for the displayed reason in “Appendix A Reasons for user login failures and abnormal logouts” and troubleshoot according to the corresponding solution.

If you cannot use the display aaa offline-record command to obtain the user offline reasons, proceed with the next step.

4. Troubleshoot the issue based on the RADIUS debugging information.

If you cannot obtain the failure reasons in the preceding steps, execute the debugging radius all command in user view on the campus BRAS to enable debugging for RADIUS. Troubleshoot the issue according to the Reply-Message field in the debugging information.

The Reply-Message field displays the PPPoE agency failure reason. Search for the displayed reason in “Appendix A Reasons for user login failures and abnormal logouts” and troubleshoot according to the corresponding solution.

5. Identify whether the campus BRAS has received agency requests from the campus AAA server.

Execute the display radius statistics command in any view on the campus BRAS to view statistics of the PPPoE agency packets between the campus BRAS and campus AAA server.

¡ If the value for the COA requests field is 0 (or the value does not change when you view this field multiple times), the campus BRAS does not receive agency requests from the campus AAA server. In this case, verify that the PPPoE agency user settings on the campus AAA server are correct to resolve the issue that the campus AAA server does not send agency requests.

¡ If the value for the COA requests field is not 0 and changes when you view this field multiple times, proceed with the next step.

6. Identify whether the campus BRAS can provide the PPPoE agency service for campus users.

Execute the display pppoe-agency packet statistics command in any view on the campus BRAS to view the negotiation packet statistics for PPPoE agency.

¡ If the value for the SEND_PADI_PKT field is 0 (or the value does not change when you view this field multiple times), the campus BRAS user does not trigger the agency process after coming online. Perform the following checks according to the PPPoE configuration guide to resolve the issue that the agency process cannot be triggered.

- Make sure the interface connecting the campus BRAS to the service provider BRAS is enabled with PPPoE agency.

- Make sure the agency group name that the campus AAA server assigns to campus BRAS user through COA messages can find the corresponding agency interface on the BRAS and the agency interface is up.

- Make sure a correct PPPoE agency forwarding policy is configured in user group view for the campus BRAS user.

¡ The campus BRAS user triggers the PPPoE agency process after coming online, but the campus BRAS does not receive the PPPoE protocol packets replied by the service provider BRAS if the following conditions exist:

- The value for the SEND_PADI_PKT field is not 0 and the value changes when you view this field multiple times.

- The value for the RECV_PADO_PKT field is 0 (or the value does not change when you view this field multiple times),

Perform the following checks according to the PPPoE configuration guide to resolve the issue that the campus BRAS cannot receive replies from the service provider BRAS.

- Make sure the interface connecting the service provider BRAS to the campus BRAS is enabled with the PPPoE server feature.

- Make sure the interface connecting the service provider BRAS to the campus BRAS is up.

¡ If the campus BRAS can send and receive PPPoE negotiation packets for PPPoE agency normally, proceed with the next step.

7. Troubleshoot on the service provider BRAS.

For the service provider BRAS, a campus PPPoE agency user is a common PPPoE user. On the service provider BRAS, troubleshoot this issue.

If the issue persists after troubleshooting, proceed with the next step.

8. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Campus user failures to access the external network on a PPPoE agency network

Symptom

On a PPPoE agency network, after a campus user that has opened a service provider agency account comes online through IPoE or PPPoE, the user can access only the campus network but cannot access the external network.

Common causes

The following are the common causes of this type of issue:

· The PPPoE agency user corresponding to a campus BRAS user is not online.

· The PPPoE agency forwarding policy configuration is incorrect on the campus BRAS.

· The PPPoE agency forwarding policy configuration is correct on the campus BRAS, but the ACL in the policy fails to be applied.

· The service provider BRAS fails.

Troubleshooting flow

Figure 4 shows the troubleshooting flowchart.

Figure 4 Flowchart for troubleshooting campus user failures to access the external network on a PPPoE agency network

Solution

1. Identify whether the PPPoE agency user corresponding to a campus BRAS user has come online successfully.

Execute the display access-user command in any view on the campus BRAS to identify whether the PPPoE agency user corresponding to the campus BRAS user has come online successfully.

¡ If the PPPoE agency user has not come online, troubleshoot this issue as described in “PPPoE user online failures and abnormal offline events.”

¡ If the PPPoE agency user comes online normally, proceed with the next step.

2. Identify whether the PPPoE agency forwarding policy configuration is correct.

Identify whether a correct ACL is specified in the pppoe-agency forward { ipv4 | ipv6 } acl { acl-number | name acl-name } command in the user group of the agency campus BRAS user on the campus BRAS.

¡ If the ACL is configured incorrectly (for example, the ACL specified in the PPPoE agency forwarding policy does not allow specifying the user-group parameter but the user-group parameter is specified in the ACL), correct the configuration.

¡ If the ACL configuration is correct, proceed with the next step.

3. Identify whether the ACL specified in the PPPoE agency forwarding policy is applied successfully.

Execute the display pppoe-agency { ipv4 | ipv6 } acl statistics command in any view on the campus BRAS to identify whether the ACL specified in the PPPoE agency forwarding policy is successfully applied.

¡ If the ACL fails to be applied, perform one of the following tasks according to the failure reason:

- If the failure reason is Hardware-count (Failed), contact Technical Support.

- If the failure reason is Hardware-count(Not enough resources to complete the operation.), execute the display qos-acl resource command in system view to collect the current ACL usage and contact Technical Support.

- If the failure reason is Hardware-count(The operation is not supported.), identify whether the software and hardware requirements of the device are met according to the product manuals. For example, identify whether the card hosting the access interface of the campus BRAS supports PPPoE agency.

¡ If the ACL is applied successfully, proceed with the next step.

4. Troubleshoot on the service provider BRAS

For the service provider BRAS, a campus PPPoE agency user is a common PPPoE user. On the service provider BRAS, troubleshoot this issue.

If the issue persists after troubleshooting, proceed with the next step.

5. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

L2TP user online failures and abnormal offline events

Symptom

An L2TP user fails to come online or abnormally goes offline.

Common causes

The following the common causes of this type of issue:

· The LAC and the LNS cannot reach each other at Layer 3.

· The service modules that establish the L2TP tunnel between the LAC and the LNS do not support L2TP.

· The LAC or the LNS is not enabled with L2TP.

· The L2TP group settings on the LAC and the LNS do not match.

· The tunnel authentication methods or authentication passwords on the LAC and the LNS are inconsistent.

· PPPoE access fails on the LAC.

· The PPP authentication methods on the LAC and the LNS are inconsistent.

· An LNS is configured with an L2TP group in LAC mode and acts as a Layer 2 tunnel switch (LTS).

· The IP address pool is configured incorrectly, and the L2TP user is not assigned an IP address.

Troubleshooting flow

Figure 5 shows the troubleshooting flowchart.

Figure 5 Flowchart for troubleshooting L2TP user online failures and abnormal offline events

Solution

1. Check whether PPPoE access services are correct on the LAC.

For more information, see "PPPoE user online failures and abnormal offline events."

If PPPoE access services are correct, proceed to the next step.

2. Identify the online failure reason and offline reason on the LNS.

¡ Use the display aaa online-fail-record command to identify the online failure reason. The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting.

¡ If you cannot obtain the online failure reasons for a user in step 1, the user might come online successfully and then go offline. In this case, use the display aaa offline-record command to display user offline records. If you cannot use the display aaa offline-record command to obtain the user offline reasons, proceed to the next step.

3. Check whether the LNS can be pinged from the LAC.

¡ If yes, proceed to the next step.

¡ If not, solve the connectivity issue.

4. Use the display device command on the LAC and the LNS to check whether the service modules used to establish the L2TP tunnel support L2TP.

¡ If yes, proceed to the next step.

¡ If not, evaluate whether the service modules can be replaced. If the issue persists after the service modules are replaced, proceed to the next step.

5. Use the display current-configuration command on the LAC and the LNS to check whether L2TP is enabled.

¡ If yes (the l2tp enable field is displayed), proceed to the next step.

¡ If not (the l2tp enable field is not displayed), use the l2tp enable command to enable L2TP. If the issue persists after L2TP is enabled, proceed to the next step.

6. Check whether the L2TP parameters in the L2TP group are configured correctly on the LAC and the LNS.

¡ On the LAC, use the display l2tp-group verbose command to check whether the LNS IP address (LNS IP field) is the same as the actual LNS IP address. If not, use the lns-ip command to change the LNS IP address.

¡ On the LNS, use the display l2tp-group verbose command to check the following items:

- Verify that the remote tunnel name is the same as the tunnel name configured on the LAC.

- Verify that the local tunnel IP address is the same as the IP address configured by the lns-ip command on the LAC.

If the issue persists after all L2TP parameters in the L2TP group are configured correctly, proceed to the next step.

7. Use the display l2tp-group verbose command on the LAC and the LNS to check whether the tunnel authentication settings are the same.

¡ Check whether the tunnel authentication states (Tunnel auth field) on the LAC and the LNS are the same. If not, use the tunnel authentication command to change the tunnel authentication status on the LAC or the LNS.

¡ If both the LAC and the LNS are enabled with tunnel authentication, verify that the tunnel authentication passwords configured on the LAC and the LNS are the same. To change the tunnel authentication password, use the tunnel password command.

¡ If the issue persists after the authentication settings are configured correctly, proceed to the next step.

8. Use the display current-configuration interface virtual-template command on the LAC and the LNS to check whether the PPP authentication methods (ppp authentication-mode field) are the same.

¡ If not, use the ppp authentication-mode command in VT interface view to configure the PPP authentication method.

¡ If yes, proceed to the next step.

9. Check whether an LAC-mode L2TP group has the same user configuration as an L2TP group configured on the LAC.

¡ If not, proceed to the next step.

¡ If yes, execute the undo user command to delete the configuration. If the issue persists after the configuration is deleted, proceed to the next step.

10. Check whether the user has been assigned an IP address.

¡ If not, configure a correct address pool on the LNS.

¡ If yes, proceed to the next step.

11. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

IPoE user online failures and abnormal offline events

This section describes the common troubleshooting method for IPoE users. For the more specific troubleshooting methods for IPoE DHCP users, IPoE NDRS users, IPoE static users, and IPoE Web users, see their respective sections.

Symptom

An IPoE user fails to come online or abnormally goes offline.

Common causes

The following the common causes of this type of issue:

· The authentication domain is configured incorrectly, which leads to authentication failure.

· The IP address pool or DHCP server is configured incorrectly, which causes the user to fail to obtain an IP address.

Troubleshooting flow

Figure 6 shows the troubleshooting flowchart.

Figure 6 Flowchart for troubleshooting IPoE user online failures and abnormal offline events

Solution

1. Use the display aaa online-fail-record command to identify the online failure reason.

<Sysname> display aaa online-fail-record username aaa

Username: aaa

Domain: test

MAC address: 0010-9400-0007

Access type: IPoE

Access interface: Ten-GigabitEthernet3/1/1

SVLAN/CVLAN: -/-

IP address: -

IPv6 address: -

Online request time: 2019/09/23 14:57:06

Online failure reason: DHCP with server no response

The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting. Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

2. Use the display aaa offline-record command to identify the offline reason.

Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

If you cannot use the display aaa offline-record command to obtain the user offline reasons, proceed with the next step.

3. Check whether the user has passed authentication.

¡ If not, examine the authentication domain configuration based on the IPoE authentication method.

¡ If yes, proceed to the next step.

4. Check whether the user has obtained an IP address.

¡ If not, examine the IP address pool or DHCP server configuration (for example, whether the DHCP service is enabled).

¡ If yes, proceed to the next step.

5. Enable service tracing to troubleshoot the issue.

6. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

IPoE DHCP user online failures and abnormal offline events

Symptom

An IPoE DHCP user fails to come online or abnormally goes offline.

Common causes

The following the common causes of this type of issue:

· Configuration errors exist. For example, the managed address configuration flag (M) is set to 0 for DHCPv6 users on an interface.

· User authentication fails.

· The user is logged out after coming online due to reasons such as timeout.

· The user is blocked.

· No DHCP messages are received.

Troubleshooting flow

Figure 7 shows the troubleshooting flowchart.

Figure 7 Flowchart for troubleshooting IPoE DHCP user online failures and abnormal offline events

Solution

1. Use the display aaa online-fail-record command to identify the online failure reason.

<Sysname> display aaa online-fail-record

Total count: 108

Username: 001094500021

Domain: dm1

MAC address: 0010-9450-0021

Access type: IPoE

Access UP ID: 1354

Access interface: XGE3/1/1

SVLAN/CVLAN: -/-

IP address: -

IPv6 address: -

Online request time: 2021/08/15 07:38:15

Online failure reason: DHCP with server no response

The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting. Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

2. Use the display aaa offline-record command to identify the offline reason.

<Sysname> display aaa offline-record

Total count: 4

Username: 001094500021

Domain: dm1

MAC address: 0010-9450-0021

Access type: IPoE

Access UP ID: 1354

Access interface: XGE3/1/1

SVLAN/CVLAN: -/-

IP address: 9.0.3.1

IPv6 address: -

Online request time: 2021/08/15 08:05:17

Offline time: 2021/08/15 08:09:08

Offline reason: DHCP release

Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

If you cannot use the display aaa offline-record command to obtain the user offline reasons, proceed with the next step.

If the offline reason cannot be identified, proceed to the next step.

3. Check whether the IPoE DHCP settings are correct.

Troubleshoot the settings according to the manuals for BRASs. For example, see tasks at a glance or configuration examples in the corresponding manuals.

¡ If configuration errors exist, correct the configuration and then try to come online again.

¡ If the configuration is correct but the issue persists, proceed to the next step.

4. Check whether the user has been blocked.

¡ Use the display ip subscriber chasten user quiet command to check whether the user has been blocked by the quiet timer. If yes, wait for the quiet timer to expire.

¡ Use the display dhcp interface-rate-suppression command to check whether the user has been suppressed by interface-based DHCP attack suppression. If the State field is Restrain, the user is suppressed. In this case, use the interface-rate-suppression threshold command to modify the DHCP packet rate threshold.

If the user is not blocked, proceed to the next step.

5. Use the display dhcp-access packet statistics command to check whether the DHCP module receives packets.

<Sysname> display dhcp-access packet statistics

Received packets

Received from clients : 32

DHCPDISCOVER : 24

DHCPREQUEST : 4

DHCPDECLINE : 0

DHCPRELEASE : 4

DHCPINFORM : 0

Received from servers : 8

DHCPOFFER : 4

DHCPACK : 4

DHCPNAK : 0

Sent packets

Send to clients : 8

DHCPOFFER : 4

DHCPACK : 4

DHCPNAK : 0

Send to servers : 148135

DHCPDISCOVER : 148127

DHCPREQUEST : 4

DHCPDECLINE : 0

DHCPRELEASE : 4

In the sample output, the count of the DHCPDISCOVER field increases, which indicates that the device receives DHCP-DISCOVER messages. In this case, execute the following commands and collect service tracing messages.

¡ Execute the trace access-user command to create a service tracing object.

¡ Execute the debugging dhcp server packet command to enable DHCP protocol message debugging.

¡ Execute the terminal debugging and terminal monitor commands to enable output of debugging messages to the current terminal and enable log output to the current terminal.

If the count of the DHCPDISCOVER field does not increase, execute the debugging ip subscriber all command to enable IPoE debugging. If the IPoE module receives DHCP-DISCOVER messages but drops them, analyze the reason according to the debug information. If the IPoE module does not receive DHCP-DISCOVER messages, proceed to the next step.

6. Check whether the device receives user messages.

On a CUPS network, for information about how to identify whether the user packets are sent to the BRAS, see "Troubleshooting issues specific to a CUPS network."

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal rxtx packet statistic slot 3 cpu 0

Net port packet loss count:

code counter

Rx packets statistic:

counter success rate

NET ->RXTX : 3177780 3177780 9 pps

Cpu code input list:(Mgment to L1 queue)

code counter success(whitelist/normal)

5 2057 2057(0/2057)

6 2077 2077(0/2077)

17 98 98(0/98)

18 48 48(0/48)

30 2091197 2091197(0/2091197)

35 573 573(0/573)

43 565 565(0/565)

45 4327 4327(0/4327)

49 79488 79488(0/79488)

50 85 85(0/85)

53 69830 69830(69823/7)

54 46567 46567(46566/1)

57 161707 161707(0/161707)

59 13052 13052(13044/8)

60 26280 26280(13953/12327)

61 30 30(0/30)

153 593518 593518(593513/5)

185 4354 4354(0/4354)

194 81927 81927(0/81927)

Callback function packets statistic:

total(r) success(r) total(c) success(c)

MACL: 0 0 0 0

NATL: 0 0 0 0

BFD: 0 0 0 0

(null): 0 0 0 0

Task input pkt statistics:

Task name total success

Main Task : 1086583 1086583

Icmp Task : 0 0

Cpu code input list:(L2 queue to platform)

code counter success drop rate

5 2057 2057 0 0

6 2077 2077 0 0

17 98 98 0 0

18 48 48 0 0

35 573 573 0 0

43 565 565 0 0

45 4327 4327 0 0

49 79488 79488 0 0

50 85 85 0 0

53 69830 69830 0 0

54 46567 46567 0 0

57 161707 161707 0 0

59 13052 13052 0 0

60 26280 26280 0 0

61 30 30 0 0

153 593518 593518 0 1

185 4354 4354 0 0

194 81927 81927 0 0

Cpu code to protocol:

5 ARP_REQ_LOCAL

6 ARP_REL

17 ARP_REQ

18 ARP_REQ_PROXY

30 DIAG

35 ND_NA

43 ND_NS

45 ND_RA

49 OSPF_HELLO,OSPF_LSU,OSPF_LSACK

50 OSPF_DD,OSPF_LSR

53 LDP_HELLO

54 LDP_NOTIF,LDP_INIT,LDP_KPALV,LDP_ADDR,LDP_LABEL

57 ISIS

59 BGP

60 BGP4P_IPV6

61 DHCP_IPOE,DHCP_SNOOPING,DHCP,DHCPv6_RELAY,DHCPv6_RELS,DHCPv6_SERV

153 IP_VSRP

185 VXLAN_GPE

194 CUSP

Debug packets statistic:

counter counter rate

NET->RXTX->SERVICE: 0 0 0 pps

SERVICE->RXTX->NET: 0 0 0 pps

failed

MbufTrSend: 0

FoundIfindex: 0

SaveCoreSta: 0

MainCoreSta: 0

TxFailedSta: 0

The 61 field represents DHCP_IPOE, DHCP_SNOOPING, and DHCP. If the received packet count for 61 increases, it means that the device has received DHCP messages and sent them to the platform. You can use debugging for the forwarding function to identify the layer on which packets are dropped step by step. If the count does not increase, execute the display hardware internal np pktcnt drop command to identify whether the driver has packet drop count.

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal np pktcnt drop slot 3 (the command for viewing the packet count varies by device model)

Current Mcode Type: SIRIUS_RELEASE

The NP 0 is Both NP

Drop packet statistics

32B7 116497 TOPparse total discarded pkts

350F 916677 TOPresolve total discarded pkts

51A 66 PRS Ingress route interface deny L2 forward

56B 384 PRS Ingress Route interface deny L2 forward

63C 403633 RSV Ingress ARP packet FTN or BROADCAST table no ma

tch

63E 372789 RSV Ingress PROTOCOL_MAC and BROADCAST table no mat

641 161878 RSV Ingress PROTOCOL_MAC.THB is set, but BROADCAST

table no match

645 149489 RSV Ingress multicast, MULTICAST.DROP is set

646 144150 RSV Ingress multicast, match MULTICAST default entr

y, but BROADCAST table no match

663 4 RSV Ingress broadcast packets from route port, PROT

OCOL_PORT table no match

If the packet drop count keeps increasing, analyze the possible issues according to the packet drop reasons.

If the packet drop count does not increase and the number of packets sent to the CPU also does not increase, it means that packets are not successfully sent to the BRAS. In this case, proceed to the next step.

7. Check the device failures.

¡ Make sure the physical connections of the device are correct.

¡ Make sure the network configuration is correct.

8. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

IPoE NDRS user online failures and abnormal offline events

Symptom

An IPoE NDRS user fails to come online or abnormally goes offline.

Common causes

The following the common causes of this type of issue:

· Configuration errors exist. For example:

¡ The access interface is not enabled with IPv6.

¡ The IPoE access mode is incorrect.

¡ No IPv6 prefix is authorized.

¡ The ND prefix pool is incorrect.

· User authentication fails.

· The user is blocked.

· No user packets are received.

Troubleshooting flow

Figure 8 shows the troubleshooting flowchart

Figure 8 Flowchart for troubleshooting IPoE NDRS user online failures and abnormal offline events

Solution

1. Use the display aaa online-fail-record command to identify the online failure reason.

<Sysname> display aaa online-fail-record

Username: user1

Domain: dm1

MAC address: 0000-5e00-01cc

Access type: IPoE

Access UP ID: 1353

Access interface: XGE3/1/1

SVLAN/CVLAN: -/-

IP address: -

IPv6 address: -

Online request time: 2021/08/15 06:09:54

Online failure reason: No prefix available

The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting. Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

You can resolve the issues caused by some failure reasons (for example, Authentication method error, Local authentication request was rejected, or No prefix available) by checking the configuration. If you cannot see the failure records for some failure reasons, proceed to the next step.

2. Use the display aaa offline-record command to identify the offline reason.

Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

If you cannot use the display aaa offline-record command to obtain the user offline reasons, proceed with the next step.

3. Check whether the IPoE NDRS user settings are correct.

Troubleshoot the settings according to the manuals for BRASs. For example, see tasks at a glance or configuration examples in the corresponding manuals.

¡ If configuration errors exist, correct the configuration and then try to come online again.

¡ If the configuration is correct but the issue persists, proceed to the next step.

4. Execute the display ppp chasten user command to identify whether the user is blocked by IPoE.

If the user is blocked, redial after the remaining blocking time expires according to the command output. If the user is not blocked, proceed to the next step.

5. Check whether the UCM and IPoE modules receive packets.

Execute the following commands for troubleshooting and collect service trace messages:

¡ Execute the trace access-user command to create a service tracing object.

¡ Execute the debugging ip subscriber all command to enable IPoE debugging.

¡ Execute the terminal debugging and terminal monitor commands to enable output of debugging messages to the current terminal and enable log output to the current terminal.

If no packets are received, proceed to the next step.

6. Check whether the BRAS receives user packets.

On a CUPS network, for information about how to identify whether the user packets are sent to the BRAS, see "Troubleshooting issues specific to a CUPS network."

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal rxtx packet statistic slot 3 cpu 0

Net port packet loss count:

code counter

Rx packets statistic:

counter success rate

NET ->RXTX : 3177780 3177780 9 pps

Cpu code input list:(Mgment to L1 queue)

code counter success(whitelist/normal)

5 2057 2057(0/2057)

6 2077 2077(0/2077)

17 98 98(0/98)

18 48 48(0/48)

30 2091197 2091197(0/2091197)

35 573 573(0/573)

43 565 565(0/565)

45 4327 4327(0/4327)

49 79488 79488(0/79488)

50 85 85(0/85)

53 69830 69830(69823/7)

54 46567 46567(46566/1)

57 161707 161707(0/161707)

59 13052 13052(13044/8)

60 26280 26280(13953/12327)

61 30 30(0/30)

153 593518 593518(593513/5)

185 4354 4354(0/4354)

194 81927 81927(0/81927)

Callback function packets statistic:

total(r) success(r) total(c) success(c)

MACL: 0 0 0 0

NATL: 0 0 0 0

BFD: 0 0 0 0

(null): 0 0 0 0

Task input pkt statistics:

Task name total success

Main Task : 1086583 1086583

Icmp Task : 0 0

Cpu code input list:(L2 queue to platform)

code counter success drop rate

5 2057 2057 0 0

6 2077 2077 0 0

17 98 98 0 0

18 48 48 0 0

35 573 573 0 0

43 565 565 0 0

45 4327 4327 0 0

49 79488 79488 0 0

50 85 85 0 0

53 69830 69830 0 0

54 46567 46567 0 0

57 161707 161707 0 0

59 13052 13052 0 0

60 26280 26280 0 0

61 30 30 0 0

153 593518 593518 0 1

185 4354 4354 0 0

194 81927 81927 0 0

Cpu code to protocol:

5 ARP_REQ_LOCAL

6 ARP_REL

17 ARP_REQ

18 ARP_REQ_PROXY

30 DIAG

35 ND_NA

43 ND_NS

45 ND_RA

49 OSPF_HELLO,OSPF_LSU,OSPF_LSACK

50 OSPF_DD,OSPF_LSR

53 LDP_HELLO

54 LDP_NOTIF,LDP_INIT,LDP_KPALV,LDP_ADDR,LDP_LABEL

57 ISIS

59 BGP

60 BGP4P_IPV6

61 DHCP_IPOE,DHCP_SNOOPING,DHCP,DHCPv6_RELAY,DHCPv6_RELS,DHCPv6_SERV

153 IP_VSRP

185 VXLAN_GPE

194 CUSP

Debug packets statistic:

counter counter rate

NET->RXTX->SERVICE: 0 0 0 pps

SERVICE->RXTX->NET: 0 0 0 pps

failed

MbufTrSend: 0

FoundIfindex: 0

SaveCoreSta: 0

MainCoreSta: 0

TxFailedSta: 0

If the received packet counts increase, it means that the device has received ARP, ND, or unknown IP packets and sent them to the platform. You can use debugging for the forwarding function to identify the layer on which packets are dropped step by step. If the counts do not increase, execute the display hardware internal np pktcnt drop command to identify whether the driver has packet drop count.

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal np pktcnt drop slot 3 (the command for viewing the packet count varies by device model)

Current Mcode Type: SIRIUS_RELEASE

The NP 0 is Both NP

Drop packet statistics

32B7 116497 TOPparse total discarded pkts

350F 916677 TOPresolve total discarded pkts

51A 66 PRS Ingress route interface deny L2 forward

56B 384 PRS Ingress Route interface deny L2 forward

63C 403633 RSV Ingress ARP packet FTN or BROADCAST table no ma

tch

63E 372789 RSV Ingress PROTOCOL_MAC and BROADCAST table no mat

641 161878 RSV Ingress PROTOCOL_MAC.THB is set, but BROADCAST

table no match

645 149489 RSV Ingress multicast, MULTICAST.DROP is set

646 144150 RSV Ingress multicast, match MULTICAST default entr

y, but BROADCAST table no match

663 4 RSV Ingress broadcast packets from route port, PROT

OCOL_PORT table no match

If the dropped packet count keeps increasing, analyze the possible issues according to the packet drop reasons.

If the dropped packet count does not increase and the number of packets sent to the CPU also does not increase, it means that packets are not successfully sent to the BRAS. In this case, proceed to the next step.

7. Check the device failures.

¡ Make sure the physical connections of the device are correct.

¡ Make sure the network configuration is correct.

8. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

IPoE static user online failure or abnormal offline event

Symptom

An IPoE static user fails to come online or abnormally goes offline.

Common causes

The following are the common causes of this type of issue:

· Incorrect user settings.

· The IP address of the user is assigned dynamically to another user.

· Authentication failure.

· The user is blocked.

· The user packets fail to be sent to the BRAS device.

Troubleshooting flow

Figure 9 shows the troubleshooting flowchart.

Figure 9 Flowchart for troubleshooting IPoE static user online failure or abnormal offline event

Solution

1. View the reason causing online failure of the IPoE static user.

Execute the display aaa online-fail-record command to display the user online failure reason.

<Sysname> display aaa online-fail-record

Username:

Domain:

MAC address: 0000-5e00-01cc

Access type: IPoE

Access UP ID: 1353

Access interface: XGE3/1/1

SVLAN/CVLAN: -/-

IP address: 2.2.2.9

IPv6 address: -

Online request time: 2021/08/15 06:09:54

Online failure reason: static user not config

The Online failure reason field in the command output displays the user online failure reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting. Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

You can resolve the issues caused by some failure reasons such as Authentication method error, Local authentication request was rejected, and Static user not config by correcting the configuration. If you cannot see the reason for the failure, proceed with the next step.

2. View the IPoE user offline reason.

If you cannot obtain the online failure reason for the user in step 1, the user might come online successfully and then go offline. In this case, use the display aaa offline-record command to display user offline records.

If the user first comes online successfully and then goes offline, the Offline reason field in the command output displays the offline reason. You can roughly locate the fault based on the failure reason, which provides guidance for later troubleshooting.

Search for the displayed reason in "Appendix A Reasons for user login failures and abnormal logouts" and troubleshoot according to the corresponding solution.

If you cannot use the display aaa offline-record command to obtain the user offline reason, proceed with the next step.

3. Verify that the IPoE static user settings are correct.

Check the IPoE static user settings by referring to the manuals for the BRAS product. For example, view tasks at a glance and configuration examples in the configuration guide for the related module.

¡ If there are incorrect settings, correct the settings and then try to come online again.

¡ If the settings are correct but the issue persists, proceed with the next step.

4. Identify whether the user has been blocked.

Execute the display ip subscriber chasten user quiet command to identify whether the user has been blocked.

¡ If the user has been blocked, redial after the remaining blocking time expires.

¡ If the user is not blocked, identify whether protocol packet loss occurs during packet transmission to the related modules on the BRAS device.

5. Verify that related modules have received packets from the user.

¡ If the static user uses unclassified-IP packet initiation, execute the debugging ip subscriber packet command to enable IPoE packet receipt and transmit debugging and troubleshoot based on debugging information.

¡ If the static user uses ARP packet initiation, execute the debugging arp packet interface ten-gigabitethernet xxx command to enable ARP packet receipt and transmit debugging and troubleshoot based on debugging information.

¡ If the static user uses ND packet initiation, execute the debugging ipv6 nd packet interface ten-gigabitethernet xxx command to enable ND packet receipt and transmit debugging and troubleshoot based on debugging information.

¡ Execute the following commands to enable service tracing messages, and collect service tracing messages and troubleshoot based on the messages.

- Execute the trace access-user command to to create a service tracing object.

- Execute the debugging ip subscriber all command to enable IPoE debugging.

- Execute the terminal debugging and terminal monitor commands to enable output of debugging messages to the current terminal and enable log output to the current terminal.

If the related modules have not received packets from the user, proceed with the next step.

6. Verify that the user packets have been sent to the BRAS device.

Execute the display hardware internal rxtx packet statistic command in probe view to view statistics about packets transmitted and received on the device driver (non-vBRAS-CPs)

For information about how to identify whether user packets have been sent to the BRAS device in a CUPS network, see "User online failure" in Troubleshooting issues specific to a CUPS network

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal rxtx packet statistic slot 3 cpu 0

Net port packet loss count:

code counter

Rx packets statistic:

counter success rate

NET ->RXTX : 3177780 3177780 9 pps

Cpu code input list:(Mgment to L1 queue)

code counter success(whitelist/normal)

5 2057 2057(0/2057)

6 2077 2077(0/2077)

17 98 98(0/98)

18 48 48(0/48)

30 2091197 2091197(0/2091197)

35 573 573(0/573)

43 565 565(0/565)

45 4327 4327(0/4327)

49 79488 79488(0/79488)

50 85 85(0/85)

53 69830 69830(69823/7)

54 46567 46567(46566/1)

57 161707 161707(0/161707)

59 13052 13052(13044/8)

60 26280 26280(13953/12327)

61 30 30(0/30)

153 593518 593518(593513/5)

185 4354 4354(0/4354)

194 81927 81927(0/81927)

Callback function packets statistic:

total(r) success(r) total(c) success(c)

MACL: 0 0 0 0

NATL: 0 0 0 0

BFD: 0 0 0 0

(null): 0 0 0 0

Task input pkt statistics:

Task name total success

Main Task : 1086583 1086583

Icmp Task : 0 0

Cpu code input list:(L2 queue to platform)

code counter success drop rate

5 2057 2057 0 0

6 2077 2077 0 0

17 98 98 0 0

18 48 48 0 0

35 573 573 0 0

43 565 565 0 0

45 4327 4327 0 0

49 79488 79488 0 0

50 85 85 0 0

53 69830 69830 0 0

54 46567 46567 0 0

57 161707 161707 0 0

59 13052 13052 0 0

60 26280 26280 0 0

61 30 30 0 0

153 593518 593518 0 1

185 4354 4354 0 0

194 81927 81927 0 0

Cpu code to protocol:

5 ARP_REQ_LOCAL

6 ARP_REL

17 ARP_REQ

18 ARP_REQ_PROXY

30 DIAG

35 ND_NA

43 ND_NS

45 ND_RA

49 OSPF_HELLO,OSPF_LSU,OSPF_LSACK

50 OSPF_DD,OSPF_LSR

53 LDP_HELLO

54 LDP_NOTIF,LDP_INIT,LDP_KPALV,LDP_ADDR,LDP_LABEL

57 ISIS

59 BGP

60 BGP4P_IPV6

61 DHCP_IPOE,DHCP_SNOOPING,DHCP,DHCPv6_RELAY,DHCPv6_RELS,DHCPv6_SERV

153 IP_VSRP

185 VXLAN_GPE

194 CUSP

Debug packets statistic:

counter counter rate

NET->RXTX->SERVICE: 0 0 0 pps

SERVICE->RXTX->NET: 0 0 0 pps

failed

MbufTrSend: 0

FoundIfindex: 0

SaveCoreSta: 0

MainCoreSta: 0

TxFailedSta: 0

If the received ARP, ND, or unclassified IP packet count has increased, the device has received the packets and sent them to the platform. You can use debugging for the forwarding function to check the layer on which packets are dropped step by step. If the count does not increase, execute the display hardware internal np pktcnt drop command to identify whether the driver has dropped packets.

<Sysname> system-view

[Sysname-probe] probe

[Sysname-probe] display hardware internal np pktcnt drop slot 3 (the command for viewing the packet count varies by device model)

Current Mcode Type: SIRIUS_RELEASE

The NP 0 is Both NP

Drop packet statistics

32B7 116497 TOPparse total discarded pkts

350F 916677 TOPresolve total discarded pkts

51A 66 PRS Ingress route interface deny L2 forward

56B 384 PRS Ingress Route interface deny L2 forward

63C 403633 RSV Ingress ARP packet FTN or BROADCAST table no ma

tch

63E 372789 RSV Ingress PROTOCOL_MAC and BROADCAST table no mat

641 161878 RSV Ingress PROTOCOL_MAC.THB is set, but BROADCAST

table no match

645 149489 RSV Ingress multicast, MULTICAST.DROP is set

646 144150 RSV Ingress multicast, match MULTICAST default entr

y, but BROADCAST table no match

663 4 RSV Ingress broadcast packets from route port, PROT

OCOL_PORT table no match

¡ If the dropped packet count keeps increasing, analyze the possible issues according to the packet drop reasons.

¡ If the dropped packet count does not increase and the number of packets sent to the CPU also does not increase, packets are not successfully sent to the BRAS. In this case, proceed with the next step.

7. Check whether faults are present on the device.

¡ Make sure the physical connections of the device are correct.

¡ Make sure the network settings of the device are correct.

8. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

IPoE Web user online failure

Web authentication page not showing up

Symptom

When an IPoE user accesses the Web authentication page or access another page than the Web authentication page, the Web authentication page does not show up.

Common causes

The following are the common causes of this type of issue:

· The Web authentication page URL is configured incorrectly in preauthentication view.

· The QoS policy is configured incorrectly in the preauthentication phase.

· Disconnectivity between the host, server, and device.

· HTTP proxy has been enabled in the browser.

· The URL entered by the user uses a non-standard TCP port number.

· An issue has occurred on the intermediate network or DNS server.

· HTTPS redirection on the device is abnormal.

· The HTTPS website accessed by the user has enabled with HTTP Strict Transport Security (HSTS).

· The portal server cannot recognize URL escape codes.

· Portal server configuration error.

Troubleshooting flow

Figure 10 shows the troubleshooting flowchart.

Figure 10 Flowchart for troubleshooting Web authentication page not showing up

Solution

1. Verify that the user has passed preauthentication.

If the user fails to pass preauthentication, resolve the issues to ensure that it can pass preauthentication.

2. Verify that Web authentication settings are correct.

¡ Verify that the IP address of the portal authentication server is configured correctly on the BRAS device.

¡ Verify that the Web authentication page URL is configured correctly on the BRAS device.

¡ Verify that the QoS policy settings for preauthentication are configured correctly on the BRAS device:

- Inbound direction: Allow packets with the portal server as the destination address to pass through.

- Outbound direction: Allow packets with the portal server as the source address to pass through.

¡ Verify that the device has been bound to an IP address group on the portal server.

¡ Verify that the endpoint IP address is within the range of the IP address group configured on the Portal server.

3. Verify that the route settings on the endpoint and the portal server are correct.

a. Disable firewall on the endpoint and ping the portal server from the endpoint. If the ping operation fails, first identify whether the route settings on the endpoint and portal server are correct, and then check the following items:

- Whether the route from the portal server to the endpoint is configured correctly.

- Whether multiple network cards exist on the endpoint and portal server.

If multiple network cards exist, not all traffic between the endpoint and the server will go through the network with portal authentication. Check specific route information and determine from which network cards the Web access traffic goes out. For example, if you are using a Windows endpoint, execute the route print command in the CLI to view specific routing information.

b. Perform ping operations by hop. First ping the gateway from the endpoint (authentication must be disabled first), and then ping the server from the gateway.

4. Identify whether HTTP proxy is enabled in the endpoint browser.

Enabling HTTP proxy in the browser will prevent users from accessing the portal authentication page. You must disable HTTP proxy in the browser. For example, to disable HTTP proxy in a Windows IE browser, click Tools, select Internet Options > Connections > LAN Settings > Proxy Server, and then turn off HTTP proxy.

5. Identify whether the entered URL uses a non-standard TCP port.

Non-standard TCP ports are non-80 or non-443 ports. If the URL entered by the user contains a non-standard TCP port, for example, http://10.1.1.1:18008, the portal authentication page will not pop up. For an HTTP URL, use port number 80. For an HTTPS URL, use port number 443.

6. Identify whether there are any issues with the intermediate network or DNS server.

¡ Identify whether the DNS server IP address is allowed on the device.

¡ Identify the intermediate network connectivity and determine whether a fault has occurred on the DNS server. On the gateway, collect traffic statistics on the downlink interface connecting the endpoint and uplink interface connecting the DNS server or mirror the endpoint messages that access the DNS server. Determine whether the gateway has sent out DNS requests but not received a response message.

7. Identify whether HTTPS redirection is enabled.

¡ If the user is accessing an HTTPS website, the device will redirect the HTTPS request of the user. You must configure an HTTPS redirect listening port number (by using the http-redirect https-port command) on the device (non-vBRAS-CPs). Before configuring an HTTPS redirect listening port number, ensure that the port is not occupied by other services. To view the occupied TCP ports, execute the display tcp command.

¡ Identify whether an SSL server policy for HTTPS redirection exists. If no such policy exists, configure it.

8. Identify whether the HTTPS website has been enabled with HSTS.

If the HTTPS website has been enabled with HSTS, a browser must use HTTPS to access the HTTPS website, and the certificate must be valid. To redirect the HTTPS request from a browser, the device will use a self-signed certificate (the device does not have a certificate from the target website, and can only use a self-signed certificate) in the disguise of the target website to establish an SSL connection with the browser. Once the browser detects that the certificate is not trusted, HTTPS redirection fails and the portal authentication page will not pop up. This situation depends on the HSTS mandatory requirements on the website and cannot be resolved. In this case, try to access another website.

9. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

¡ Snapshots of portal related configuration on the portal server.

¡ Packet capture files for packet transmission between the device and server.

¡ Snapshots of the client browser issues.

¡ Debugging information collected by executing the debugging portal and debugging portal commands.

Access failure to the Web authentication page

Symptom

A user fails Web authentication or an authentication anomaly occurs.

Common causes

The following are the common causes of this type of issue:

· The shared key configured in portal authentication server view on the BRAS device is inconsistent with that on the portal authentication server.

· The portal authentication server address configured in portal authentication server view on the BRAS device does not exist.

· The portal messages received on the BRAS device are invalid.

· The IPS domain used by the Web user is incorrect.

· The shared key configured in RADIUS view is inconsistent with that on the RADIUS server.

· The RADIUS server rejects the authentication request.

· The RADIUS server does not respond.

Troubleshooting flow

Figure 11 shows the troubleshooting flowchart.

Figure 11 Flowchart for troubleshooting access failure to the Web authentication page

Solution

1. Identify whether the shared key configured in portal authentication server view on the BRAS device is inconsistent with that on the portal authentication server.

If a request timeout message is displayed on the Web login page after you enter the username and password on the page for coming online, the shared key configured in portal authentication server view on the BRAS device might be inconsistent with that on the portal authentication server.

Execute the debugging portal error command on the BRAS device and enable portal error debugging. If following information is generated on the device, the shared key configured on the BRAS device is inconsistent with that on the portal server.

*Jul 28 17:51:20:774 2021 Sysname PORTAL/7/ERROR: -MDC=1; Packet validity check failed due to invalid key.

If the shared key configured on the BRAS device is inconsistent with that on the portal server, change the shared key configured in portal server view on the BRAS device or the shared key configured on the portal authentication server to ensure that they are consistent.

2. Identify whether the portal authentication server IP address configured in portal authentication server view on the BRAS device exists.

When the portal server receives an authentication packet from the BRAS device, it will verify whether the source IP of the message is an allowed IP. If the IP is not allowed, the packet is considered invalid and will be discarded directly.

If a request timeout message is displayed on the Web login page after you enter the username and password on the page for coming online, the portal authentication server IP address configured in portal authentication server view on the BRAS device exist might not exist.

Execute the debugging portal error command on the BRAS device and enable portal error debugging. If following information is generated on the device, the portal authentication server IP address configured on the device is incorrect.

*Jul 28 19:15:10:665 2021 Sysname PORTAL/7/ERROR: -MDC=1;Packet source unknown. Server IP:192.168.161.188, VRF Index:0.

If the portal authentication server IP address configured on the device is incorrect, execute the ip command in portal server view to modify the IP address of the portal server.

3. Identify whether the ISP domain is configured correctly on the device.

For authentication to be performed on users, make sure the ISP domain is configured correctly on the device.

If a message that the device rejects the request is generated on the Web login page after you enter the username and password on the page for coming online, the ISP domain might be configured incorrectly.

Execute the debugging portal error command on the BRAS device and enable portal error debugging. If following information is generated on the device, the ISP domain on the device is configured incorrectly.

*Jul 28 19:49:12:725 2021 Sysname PORTAL/7/ERROR: -MDC=1; User-SM [21.0.0.21]: AAA processed authentication request and returned error.

If the ISP domain is configured incorrectly, execute the related command to change the ISP domain used by the Web user to be correct.

4. Identify whether the shared key configured in RADIUS view on the device is consistent with that configured on the RADIUS server.

If a request timeout message is displayed on the Web login page after you enter the username and password on the page for coming online, the shared key configured in RADIUS view on the device might be inconsistent with that configured on the RADIUS server..

Execute the debugging radius error command on the BRAS device and enable RADIUS error debugging. If following information is generated on the device, the shared key configured in RADIUS view on the device is inconsistent with that configured on the RADIUS server.

*Jul 28 19:49:12:725 2021 Sysname RADIUS/7/ERROR: -MDC=1; The response packet has an invalid Response Authenticator value.

When the device sends an authentication request to the RADIUS server, the server will first verify the request message using the shared key. If the verification fails, the server will notify the device that the verification has failed. If the shared key is configured incorrectly, change the key in RADIUS view on the device or the key on the RADIUS server to ensure that they are consistent.

5. Identify whether the portal packets are valid.

When the device receives a portal protocol packet from the portal server, it verifies the validity of the packet. If the packet length or the packet verification segment is incorrect, the packet will be considered invalid and discarded.

Execute the display portal packet statistics command to check whether the number of invalid packets is increasing. If the number of invalid packets is increasing, execute the debugging portal error command to enable portal error debugging for troubleshooting.

If the portal protocol packets are invalid, identify the reason causing message invalidity with the assistance of technical support personnel, resolve the issues, and make sure the portal packets are valid.

6. Identify whether the device fails to obtain user physical information.

During the user's online process, the portal module will obtain the user's physical information and determine interface and other information based on physical information. If the device fails to obtain the user's physical information, the user will fail to go online.

Execute the debugging portal event command and enable portal event debugging. If following information is generated on the device, the device fails to obtain user's physical information.

*Jul 28 19:49:12:725 2021 Sysname PORTAL/7/ERROR: -MDC=1; User-SM [21.0.0.21]: Failed to find physical info for ack_info.

If the device fails to obtain physical information of the user, identify whether an entry for the authentication user exists on the device. If no such entry exists, locate the reason.

7. Identify whether the RADIUS server rejects the authentication request.

There are various reasons why a RADIUS server rejects an authentication request. The most common ones are incorrect username or password and mismatch of the RADIUS server authorization policy. To resolve the issue, first view the authentication logs on the server side or enable RADIUS error debugging on the device to identify the root cause. Then, adjust the server, endpoint, or device configuration according to the root cause.

8. Identify whether the RADIUS server responds.

You can use the following methods to quickly identify whether the RADIUS server responds.

¡ Execute the display radius scheme command on the BRAS device to view the server status. If the status is Blocked, the server is not available.

¡ Identify whether the following message has been generated on the device.

RADIUS/4/RADIUS_AUTH_SERVER_DOWN: -MDC=1; RADIUS authentication server was

blocked: server IP=192.168.161.188, port=1812, VPN instance=public.

¡ Execute the debugging radius event command and enable event debugging for the RADIUS module. If following information is generated on the device, the RADIUS server does not respond.

*Jul 28 19:49:12:725 2021 Sysname RADIUS/7/evnet: -MDC=1; Reached the maximum retries.

If the RADIUS server does not respond, perform the following tasks:

a. Identify whether the device IP address has been added to the server.

- If not added, add the correct device IP address to the server.

- If added, determine whether the IP address of the device added to the server is consistent with the source IP address of the authentication request. The device uses the IP address of the default outgoing interface as the source IP address of the RADIUS authentication request. You can change the source IP address by using the source-ip command as needed. For more information about the source-ip command, see AAA command reference in BRAS Services Command References.

b. Identify whether the link between the device and server is normal, for example, whether the firewall between the two does permit the RADIUS packets (default authentication port: 1812). If a large number of users cannot be authenticated and a RADIUS server down log is generated on the device, the server or intermediate network might be abnormal, which must be identified one by one.

9. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

¡ Snapshots of portal related configuration on the portal server.

¡ Packet capture files for packet transmission between the device and AAA server

¡ Snapshots of the client browser issues.

¡ Debugging information collected by executing the debugging portal command.

Troubleshooting issues specific to a CUPS network

User online failure

This chapter describes how to troubleshoot issues specific to a CP and UP separation (CUPS) network. For how to troubleshoot issues on other networks, see the chapters for troubleshooting common BRAS access issues.

Symptom

A user failed to come online on a CUPS network.

Common causes

The following are the common causes of this type of issue:

· BRAS-VM registration failure.

· FWD-VM registration failure.

· NETCONF connection failure.

· CUSP channel failure.

· VXLAN tunnel failure.

· Unmanaged remote interfaces.

· Configuration deployment failure.

· Network failure.

Troubleshooting flow

Figure 12 shows the troubleshooting flowchart.

Figure 12 Flowchart for troubleshooting user online failures

Solution

1. Execute the display vm command on the CP to verify if BRAS-VMs and FWD-VMs register with the CTRL-VM system successfully.

¡ If the Registration field displays Registered, the BRAS-VMs and FWD-VMs have registered with the CTRL-VM system successfully.

¡ If the Registration field does not display Registered, the BRAS-VMs and FWD-VMs fail to register with the CTRL-VM system. For more information, see "VM registration failure."

2. Execute the display netconfc session command on the CP to verify if a NETCONF connection is established between the CP and specified UP.

¡ If you can obtain the display, a NETCONF connection has been established between the CP and UP. Please go to the next step.

¡ If you cannot obtain the display, a NETCONF connection fails to be established between the CP and UP. For how to troubleshoot the issue, see "CP-UP connection management issues."

3. Execute the display cusp controller command on the CP to verify if a CUSP channel is established between the CP and specified UP.

¡ If the Connection state field displays Established, a CUSP channel has been established. Please go to the next step.

¡ If the Connection state field does not display Established, a CUSP connection fails to be set up. For how to troubleshoot the issue, see "CP-UP connection management issues."

4. Execute the display protocol-tunnel verbose command on the CP to verify if a VXLAN tunnel is established between the CP and specified UP.

¡ If the Active field displays Yes, a VXLAN tunnel has been established. Please go to the next step.

¡ If the Active field displays No, a VXLAN channel fails to be established. For how to troubleshoot the issue, see "CP-UP connection management issues."

5. Verify if the CP has deployed required BRAS configuration to the specified UP.

Execute the display this command on the user access interface on the UP to verify if cp-management configuration exists on the interface.

¡ If the interface has cp-management configuration, the interface has been remotely managed by the CP, which indicates that BRAS configuration has been deployed correctly.

¡ If the interface does not have cp-management configuration, go to the next step.

6. Verify if the UP has received the request packet for coming online.

Execute the display protocol-tunnel packet statistics command on the UP to obtain the outbound protocol packet statistics.

¡ If the number of the corresponding packets increases, go to the next step.

¡ If the number of the corresponding packets does not change, execute the debugging ucm forward all command to enable all types of debugging functions for the UCM forwarding plane.

- If the system outputs debugging information, the UP has received the packet. Please contact Technical Support.

- If the system does not output any debugging information, the UP does not receive the packet. Please check the network configuration and links.

7. Verify if the CP has received the request packet for coming online.

Execute the display protocol-tunnel packet statistics command on the CP to obtain the inbound protocol packet statistics.

¡ If the number of the corresponding packets increases, go to the next step.

¡ If the number of the corresponding packets does not change, capture packets traversing the NIC that connects to the CP by using Tcpdump on the UP.

¡ If the packet has been sent to the CP, capture packets on an interface of each FWD-VM by using the packet capture feature. Verify if the packet has been set to a FWD-VM. If a FWD-VM has received the packet, execute the display driver ten-gigabitethernet xxx message command in probe view to obtain the packet drop statistics for the x86 driver. The packet might be dropped due to incorrect VLAN ID. To resolve the issue, re-create the VXLAN tunnel and come online again.

8. Troubleshoot the issue as described in the chapters about troubleshooting PPPoE, L2TP, or IPoE user online failures.

9. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

CP-UP connection management issues

CUPS channel failure

Symptom

On a CUPS system, the control channel, management channel, or protocol channel between the CP and a specified UP is abnormal. When you execute the cudetect cu tunnel-state command on the CTRL-VM, a minimum of one of the NETCONF Tunnel, CUSP Tunnel, and Protocol Tunnel fields does not display OK. For example:

<Sysname> cudetect cu tunnel-state up-id 1024

Please wait a few minutes...

Finished.

NETCONF Tunnel: NOK

Please configure the source IP of the NETCONF connetion abc to a interface on CP.

Please check the route to destination IP on CP.

CUSP Tunnel: OK

Protocol Tunnel: NOK

Please check the listening IP of the CUSP controller and the source IP of the protocol tunnel on CP.

Common causes

The following are the common causes of this type of issue:

· Management channel—Incorrect NETCONF connection settings.

· Control channel—Incorrect CUSP settings.

· Protocol channel—Incorrect VXLAN tunnel settings.

Troubleshooting flow

Troubleshoot this type of issue as follows:

1. Check the configuration for the management channel between the CP and UP.

2. Check the configuration for the control channel between the CP and UP.

3. Check the configuration for the protocol channel between the CP and UP.

Figure 13 shows the troubleshooting flowchart.

Figure 13 Flowchart for troubleshooting CUPS channel failures

Solution

1. Check detailed management channel configuration on the CP and UP.

# Execute the display current-configuration configuration netconf-client command on the CP to check the CP-side management channel configuration.

netconf-client

source-address 2.2.2.2

connection 1024

user-name netconf password cipher $c$3$gwdAnb/zm8CEwMs5H9eQ89Hf4JFKXw==

destination-address 1.1.1.1

# Execute the display current-configuration configuration up-manage command on the CP to check the NETCONF connection profile bound to the UP.

bind netconf-connection 1024

# Execute the display current-configuration | begin ssh command on the UP to check the UP-side management channel configuration.

ssh server enable

ssh user netconf service-type netconf authentication-type password

local-user netconf class manage

password hash

bDm4CAp6rlXr9txtlp2w0URVUj8iKJ5a6MhLHmBMoHw==

service-type ssh

authorization-attribute user-role network-admin

authorization-attribute user-role network-operator

netconf ssh server enable

# Execute the cudetect cu tunnel-state up-id up-id command in any view on the CTRL-VM to obtain the value for the NETCONF Tunnel field.

¡ If the NETCONF Tunnel field displays NOK, troubleshoot the issue based on the displayed massage, as shown in the following table.

Message	Possible causes	Recommended action
Please configure the source IP of the NETCONF connetion connetion-name to a interface on CP	No IP address is configured on a CP-side interface. The connetion-name field displays the name of a NETCONF connection profile.	Specify a loopback interface address as the source IP address used for setting up a NETCONF connection to a UP.
Please check the route to destination IP on CP	No route to the UP exists on the CP.	Configure a static route or routing protocol on the CP.
Please check the username and password on CP	The username or password configured for setting up a NETCONF connection to a UP on the CP is invalid.	Make sure the username and password configured for setting up a NETCONF connection to the UP match the local SSH user configuration on the UP. Local SSH users use the password authentication method on the UP. To configure the username and password configured for setting up a NETCONF connection to the UP, execute the user-name command in NETCONF client view on the CP.
Please check the network configuration between CP and UP	No IP address or route to the CP is configured on the UP, or the network between the CP and the UP has failed.	1. Specify an IP address for the interface used for the NETCONF connection on the UP. Make sure the IP address is the same as that specified by using the destination-address command in NETCONF client view on the CP. 2. Execute the display ip routing-table command on the UP to verify that the source IP address used for setting up a NETCONF connection to the UP (specified by using the source-address command in NETCONF client view on the CP) is reachable. If the source IP address is unreachable, configure a static route or routing protocol on the UP.
Please check the NETCONF SSH configuration between CP and UP	Errors exist in the SSH configuration on the CP and the UP.	Verify that the SSH configuration on the CP and UP is complete and correct.
Others	N/A	See "Management channel establishment failure."

¡ If the NETCONF Tunnel field displays NA, the NETCONF module is abnormal. To troubleshoot the issue, see "Management channel establishment failure."

¡ If the NETCONF Tunnel field displays OK, the management channel between the CP and UP operates correctly. Identify whether an error occurs on the other channels.

2. Check detailed control channel configuration on the CP and UP.

# Execute the display current-configuration configuration cusp-controller and display current-configuration configuration up-manage commands on the CP to check the control channel configuration on the CP and UP, respectively.

cusp controller

listening-ip 2.2.2.2

agent up1

agent-ip 1.1.1.1

up-manage id 1024

control-tunnel cusp-agent up1

up-config

cusp agent up1

local-address 1.1.1.1

controller address 2.2.2.2

# Execute the cudetect cu tunnel-state up-id up-id command in any view on the CTRL-VM to obtain the value for the CUSP Tunnel field.

¡ If the CUSP Tunnel field displays NOK, troubleshoot the issue based on the displayed massage, as shown in the following table.

Message	Possible causes	Recommended action
Please configure the CUSP controller on CP	The CUSP controller feature is not enabled on the CP.	Execute the cusp controller command in system view on the CP to enable the CUSP controller feature.
Please configure the listening IP on CP	No listening IP address is specified for the CUSP controller on the CP.	Execute the listening-ip command in CUSP controller view on the CP to specify a listening IP address for the CUSP controller.
Please configure the listening IP to an interface on CP	No listening IP address is specified for the CUSP controller on an interface on the CP.	Specify an IP address for the CP-side interface on the CUSP control channel. Make sure the IP address is the same as the listening IP address for the CUSP controller specified by using the listening-ip command in CUSP controller view on the CP.
Please configure the CUSP agent on CP	No CUSP agent is added on the CP.	Execute the agent command in agent view on the CP to create a CUSP agent.
Please configure the CUSP agent IP on CP	No CUSP agent to which a CUSP controller can connect is specified by its IP address on the CP.	Execute the agent-ip command in agent view on the CP to specify an IP address for the CUSP agent to which a CUSP controller can connect.
Please check the IP version of the listening IP and CUSP agent IP on CP	The IP version of the listening IP address of the CUSP controller on the CP is different from the IP version of the CUSP agent IP on the CP.	· Execute the listening-ip command in CUSP controller view on the CP to edit the listening IP address of the CUSP controller. · Execute the agent-ip command in agent view on the CP to edit the CUSP agent IP address.
Please configure the VPN instance on CP	No VPN instance to which a CUSP controller belongs is created on the CP.	Specify an existing VPN instance when you execute the listening-ip command in CUSP controller view on the CP.
Please check the listening IP on CP and the controller address on UP	The listening IP of the CUSP controller on the CP is different from the CUSP controller IP on the UP.	· Execute the listening-ip command in CUSP controller view on the CP to edit the listening IP address of the CUSP controller. · Execute the controller address command in CUSP agent view to edit the CUSP controller IP address.
Please check the agent IP on CP and the local address on UP	The CUSP agent IP configured on the CP is different from the local IP address of the CUSP agent on the UP.	· Execute the agent-ip command in agent view on the CP to edit the CUSP agent IP address. · Execute the local-address command in CUSP agent view to edit the local IP address of the CUSP agent.
Please configure the CUSP agent on UP	No CUSP agent is configured on the UP.	Execute the cusp agent command in UP-config view on the CP to create a CUSP agent.
Please configure the local address on UP	No local IP address is specified for a CUSP agent on the UP.	Execute the local-address command in CUSP agent view on the CP to specify a local IP address for a CUSP agent.
Please configure the controller address on UP	No CUPS controller IP address is specified for the CUSP agent on the UP.	Execute the controller address command in CUSP agent view on the CP to specify a CUSP controller IP address for the CUSP agent.
Please check the IP version of the local address and controller address on UP	The IP version of the CUSP controller IP is different from the IP version of the local IP address of the CUSP agent on the CP.	Execute the undo local-address or undo controller address command in CUSP agent view on the CP to delete the incorrect IP address configuration and reconfigure it.
Cannot check the UP configuration because of the disconnection of the CU NETCONF tunnel	The management channel between the CP and the UP is abnormal, so the CP cannot check the CUSP configuration on the UP.	Return to step 1 to check detailed management channel configuration on the CP and UP.

¡ If the CUSP Tunnel field displays NA, the error occurring on the channel is unknown. To troubleshoot the issue, see "Control channel establishment failure."

¡ If the CUSP Tunnel field displays OK, the control channel between the CP and UP operates correctly. Identify whether an error occurs on the other channels.

3. Check detailed protocol channel configuration on the CP and UP.

# Execute the display current-configuration | begin up-manage command on the CP to check the protocol channel configuration on the CP and UP.

up-manage id 1024

protocol-tunnel vxlan 10 source 2.2.2.2 destination 1.1.1.1

cu-agent

protocol-tunnel vxlan 10 source 1.1.1.1 destination 2.2.2.2

# Execute the cudetect cu tunnel-state up-id up-id command on the CTRL-VM to obtain the value for the Protocol Tunnel field.

¡ If the Protocol Tunnel field displays NOK, troubleshoot the issue based on the displayed massage, as shown in the following table.

Message	Possible causes	Recommended action
Please configure the protocol tunnel on CP	No protocol channel parameters are configured on the CP.	Execute the protocol-tunnel command in UP-manage view on the CP to configure the parameters for the protocol channel between the CP and UP.
Please check the listening IP of the CUSP controller and the source IP of the protocol tunnel on CP	The protocol channel source IP and the CUSP controller listening IP are different on the CP.	Execute the protocol-tunnel command in UP-manage view on the CP to edit the protocol channel source IP address. Make sure the protocol channel source IP address is the same as the CUSP controller listening IP address specified by using the listening-ip command.
Please check the agent IP of the CUSP controller and the destination IP of the protocol tunnel on CP	The protocol channel destination IP and the CUSP controller agent IP are different on the CP.	Execute the protocol-tunnel command in UP-manage view on the CP to edit the protocol channel destination IP address. Make sure the protocol channel destination IP address is the same as the CUSP controller agent IP address specified by using the agent-ip command.
Please check the source IP of the protocol tunnel on CP and the destination IP of the protocol tunnel on UP	The protocol channel source IP on the CP is different from the protocol channel destination IP on the UP.	· Execute the protocol-tunnel command in UP-manage view on the CP to edit the protocol channel source IP address on the CP. · Execute the protocol-tunnel command in CU agent view to edit the protocol channel destination IP address on the UP.
Please check the destination IP of the protocol tunnel on CP and the source IP of the protocol tunnel on UP	The protocol channel destination IP on the CP is different from the protocol channel source IP on the UP.	· Execute the protocol-tunnel command in UP-manage view on the CP to edit the protocol channel destination IP address on the CP. · Execute the protocol-tunnel command in CU agent view to edit the protocol channel source IP address on the UP.
Please configure the protocol tunnel on UP	No protocol channel parameters are configured on the UP.	Execute the protocol-tunnel command in CU agent view on the CP to configure the parameters for the protocol channel between the CP and UP.
Please check the local address of the CUSP agent and the source IP of the protocol tunnel on UP	The protocol channel source IP and the local IP address of the CUSP agent are different on the UP.	Execute the protocol-tunnel command in CU agent view on the CP to edit the protocol channel source IP address on the UP. Make sure the protocol channel source IP address is the same as the local IP address of the CUSP agent specified by using the local-address command.
Please check the controller address of the CUSP agent and the destination IP of the protocol tunnel on UP	The protocol channel destination IP and the controller IP of the CUSP agent are different on the UP.	Execute the protocol-tunnel command in CU agent view on the CP to edit the protocol channel destination IP address on the UP. Make sure the protocol channel destination IP address is the same as the controller IP address of the CUSP agent specified by using the controller address command.
Please check the VXLAN ID of the protocol tunnel between CP and UP	The VXLAN tunnel ID of the protocol channel is different on the CP and the UP.	· Execute the protocol-tunnel command in UP-manage view on the CP to edit the VXLAN tunnel ID on the CP. · Execute the protocol-tunnel command in CU agent view to edit the VXLAN tunnel ID on the UP.
Please check the abnormal state of the CUSP tunnel between CP and UP	The state of the control channel between the CP and the UP is abnormal.	Return to step 2 to check detailed control channel configuration on the CP and UP.
Cannot check the configuration of the protocol tunnel on UP because of the disconnection of the CU NETCONF tunnel	The management channel between the CP and the UP is abnormal, so the CP cannot check the protocol channel configuration on the UP.	Return to step 1 to check detailed management channel configuration on the CP and UP.

¡ If the Protocol Tunnel field displays NA, the VXLAN module is abnormal and the troubleshooting tool cannot detect the reason. To troubleshoot the issue, see "Protocol channel establishment failure."

¡ If the Protocol Tunnel field displays OK, the protocol channel between the CP and UP operates correctly.

4. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

None.

Log messages

None.

Management channel establishment failure

Symptom

No management channel is established between the CP and specified UP. When you execute the display netconfc session command on the CP, no NETCONF session information about the specified UP is displayed.

Common causes

The following are the common causes of this type of issue:

· Physical link failure, which causes route failure for the CP and UP.

· Management channel configuration errors on the CP or UP.

Troubleshooting flow

Figure 14 shows the troubleshooting flowchart.

Figure 14 Flowchart for troubleshooting management channel establishment failures

Solution

1. Verify if an error occurs on the physical link.

a. On the CP, ping the IP address of the interface on the UP, which is directly connected to the CP.

If the ping fails, execute the display ip routing-table or display route-static routing-table command on the CP to obtain the output interface of the route to the UP. Then, execute the display interface command to check the output interface state.

<CTRL-VM> display interface ten-gigabitethernet 1/5/0

Ten-GigabitEthernet1/5/0

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

b. If the Current state field displays Administratively DOWN, execute the undo shutdown command on the interface to bring up the interface. If the Current state field displays DOWN, check the physical connection of the interface.

c. Repeat the above steps on the UP to check and repair the output interface of the route to the CP.

d. If other devices exist between the CP and UP, repeat steps a and b on each hop to check and repair the physical interfaces connecting to other devices.

e. If the physical link between the CP and specified UP is correct but the issue persists, go to the next step.

2. Execute the display current-configuration configuration netconf-client command on the CP to check the CP-side management channel configuration.

<CTRL-VM> display current-configuration configuration netconf-client

netconf-client

source-address 2.2.2.2

connection 1024

user-name netconf password cipher $c$3$J29ZV3fWskY85w0NwEO1p/LAWauPdx6Kw4xiLOn

W2dPMGEs=

destination-address 1.1.1.1

connection 1025

user-name netconf password cipher $c$3$YhPZ2Xk+MH9BNcxshQ0w8fewibpnQw2ojT1xkP2

hax3HDaE=

destination-address 3.3.3.3

Execute the cudetect cu tunnel-state up-id up-id command on the CTRL-VM. If the NETCONF Tunnel field displays NOK or NA, check the detailed management channel configuration on the CP and UP as described in "CUPS channel failure." If the NETCONF Tunnel field displays OK but the issue persists, go to the next step.

3. Execute the display current-configuration | begin ssh command on the UP to check the UP-side management channel configuration.

<UP1024> display current-configuration | begin ssh

ssh server enable

ssh user netconf service-type netconf authentication-type password

...

local-user netconf class manage

password hash $h$6$nJfK2tYuvrbih32X$+reBw1rUDg9R3z1rJ2+cs09hYIVQT7IzzxdnZe2/Nsg

liHTsJI+qDT/dbRqLQpP+it44esvq9xRfcujMdRB9Bw==

service-type ssh

authorization-attribute user-role network-admin

authorization-attribute user-role network-operator

netconf ssh server enable

return

¡ Make sure you have enabled the Stelnet server on the UP by executing the ssh server enable command.

¡ Make sure you have enabled NETCONF over SSH on the UP by executing the netconf ssh server enable command.

4. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module: HH3C-NCM-MIB

· hh3cNcmCUConnectFailed (1.3.6.1.4.1.25506.2.201.3.0.3)

Log messages

· NCM/2/NCM_CREATE_CHANNEL_FAILED

Packet forwarding failure for the management channel

Symptom

The management channel between the CP and UP fails to forward management packets correctly. As a result, user service traffic is discarded.

Common causes

The common cause of this type of issue is physical link failure, which causes route failure for the CP and UP.

Troubleshooting flow

Figure 15 shows the troubleshooting flowchart.

Figure 15 Flowchart for troubleshooting packet forwarding failures for the management channel

Solution

1. Verify if an error occurs on the physical link.

a. On the CP, ping the IP address of the interface on the UP, which is directly connected to the CP.

<CTRL-VM> display interface ten-gigabitethernet 1/5/0

Ten-GigabitEthernet1/5/0

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

c. Repeat the above steps on the UP to check and repair the output interface of the route to the CP.

d. If other devices exist between the CP and UP, repeat steps a and b on each hop to check and repair the physical interfaces connecting to other devices.

2. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module: HH3C-NCM-MIB

· hh3cNcmCUConnDisconnected (1.3.6.1.4.1.25506.2.201.3.0.1)

Log messages

· NCM/1/NCM_SESSION_DISCONNECTED

Control channel establishment failure

Symptom

No control channel is established between the CP and UP. When you execute the display cusp controller command on the CP, no fields about the CUSP agent is displayed, such as the Agent name, UP ID, and Control tunnel state fields.

Common causes

The following are the common causes of this type of issue:

· Physical link failure, which causes route failure for the CP and UP.

· Control channel configuration errors on the CP or UP.

Troubleshooting flow

Figure 16 shows the troubleshooting flowchart.

Figure 16 Flowchart for troubleshooting control channel establishment failures

Solution

1. Verify if an error occurs on the physical link.

a. On the CP, ping the IP address of the interface on the UP, which is directly connected to the CP.

<CTRL-VM> display interface ten-gigabitethernet 1/5/0

Ten-GigabitEthernet1/5/0

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

c. Repeat the above steps on the UP to check and repair the output interface of the route to the CP.

d. If other devices exist between the CP and UP, repeat steps a and b on each hop to check and repair the physical interfaces connecting to other devices.

e. If the physical link between the CP and specified UP is correct but the issue persists, go to the next step.

2. Check the CP-side control channel configuration.

Execute the display current-configuration | begin cusp command on the CP to verify if you have executed the listening-ip and agent-ip commands.

<CTRL-VM> display current-configuration | begin cusp

cusp controller

listening-ip 2.2.2.2

agent up1024

agent-ip 1.1.1.1

agent up1025

agent-ip 3.3.3.3

...

Execute the cudetect cu tunnel-state up-id up-id command on the CTRL-VM. If the CUSP Tunnel field displays NOK or NA, check the detailed control channel configuration on the CP and UP as described in "CUPS channel failure." If the CUSP Tunnel field displays OK but the issue persists, go to the next step.

3. Execute the display current-configuration | begin cusp command on the UP to check the UP-side control channel configuration.

<UP1024> display current-configuration | begin cusp

cusp agent up1024

local-address 1.1.1.1

controller address 2.2.2.2

...

¡ Make sure the IP address specified by using the local-address command in CUSP agent view on the UP is the same as that specified by using the agent-ip command in agent view on the CP.

¡ Make sure the IP address specified by using the controller address command in CUSP agent view on the UP is the same as that specified by using the listening-ip command in CUSP controller view on the CP.

4. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module: HH3C-CUSP-MIB

· hh3cCuspServerDisconnect (1.3.6.1.4.1.25506.2.190.1.2.0.1)

· hh3cCuspClientDisconnect (1.3.6.1.4.1.25506.2.190.1.2.0.3)

Log messages

· CUSP/5/CUSP_CP_DISCONNECT

· CUSP/5/CUSP_UP_DISCONNECT

Packet forwarding failure for the control channel

Symptom

The control channel between the CP and UP failed to forward control packets correctly, causing user service traffic to be dropped.

Common causes

The common cause of this type of issue is that a physical link failure causes route unreachability between CP and UP devices.

Troubleshooting flow

Figure 17 shows the troubleshooting flowchart.

Figure 17 Flowchart for troubleshooting packet forwarding failure for the control channel

Solution

1. Check the physical link.

On the CP, ping the IP address of the interface that directly connects the UP to the CP.

If the address cannot be pinged, execute the display ip routing-table or display route-static routing-table command to identify the outgoing interface of the route to the UP. Then, execute the display interface command to view the status of the outgoing interface.

<CTRL-VM> display interface ten-gigabitethernet 1/5/0

Ten-GigabitEthernet1/5/0

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

¡ If the current state of the interface is Administratively DOWN, execute the undo shutdown command to bring up the interface. If the current state of the interface is DOWN, verify that the physical link of the interface is normal.

¡ Repeat the previous steps on the UP to check and repair the outgoing interface of the route to the CP.

¡ If other devices exist between the CP and UP, use the previous steps to check and repair the status of the connecting interfaces hop by hop.

2. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module name: HH3C-CUSP-MIB

· hh3cCuspServerDisconnect (1.3.6.1.4.1.25506.2.190.1.2.0.1)

· hh3cCuspClientDisconnect (1.3.6.1.4.1.25506.2.190.1.2.0.3)

Log messages

· CUSP/5/CUSP_CP_DISCONNECT

· CUSP/5/CUSP_UP_DISCONNECT

Protocol channel establishment failure

Symptom

On the CP and UP, execute the display protocol-tunnel verbose command. The outputs show that the VXLAN channel between the CP and UP is not established successfully (the value of the Active field is No).

Common causes

The following are the common causes of this type of issue:

· The VXLAN related settings for the protocol channel are incorrect.

· The CUSP channel between the CP and the specified UP fails.

· Physical link failure.

Troubleshooting flow

Figure 18 shows the troubleshooting flowchart.

Figure 18 Flowchart for troubleshooting protocol channel establishment failure

Solution

1. Check the physical link.

On the CP, execute the display ip routing-table or display route-static routing-table command to identify the outgoing interface of the route to the UP. Then, execute the display interface command to view the status of the outgoing interface.

<Sysname> display interface ten-gigabitethernet 1/5/0

Ten-GigabitEthernet1/5/0

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

¡ Repeat the previous steps on the UP to check and repair the outgoing interface of the route to the CP.

¡ If other devices exist between the CP and UP, use the previous steps to check and repair the status of the connecting interfaces hop by hop.

¡ If the physical link between the CP and UP is normal and the issue persists, perform the following operations.

2. Check the VXLAN related settings for the protocol channel.

On the CP, execute the display current-configuration configuration up-manage command to view the detailed configuration for the protocol channel between the CP and UP.

<Sysname> display current-configuration configuration up-manage

up-manage id 1024

protocol-tunnel vxlan 10 source 2.2.2.2 destination 1.1.1.1

cu-agent

protocol-tunnel vxlan 10 source 1.1.1.1 destination 2.2.2.2

On the CTRL-VM, execute the cudetect cu tunnel-state up-id up-id command. If the Protocol Tunnel field displays NOK or NA, check and repair the UP-CP protocol channel as described in the step for checking the protocol channel configuration in “CUPS channel failure.”

If the physical link between the CP and UP is normal and the issue persists, proceed to the following step.

3. Check the CUSP channel between the CP and the specified UP.

On the CP, execute the display cusp controller command.

¡ If no CUSP agent information (including agent name, UP ID, and control tunnel state) is displayed for the specified UP, it indicates that no CUSP channel is established. In this case, troubleshoot the issue as described in “Control channel establishment failure.”

¡ If the Connection state field displays Established, it indicates that the CUSP channel has been established successfully. Proceed to the next step.

4. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Packet forwarding failure for the protocol channel

Symptom

The protocol channel between the CP and UP failed to forward VXLAN packets correctly, causing user service traffic to be dropped.

Common causes

The following are the common causes of this type of issue:

· The VXLAN related settings for the protocol channel are incorrect.

· User access interfaces on the UP are not managed on the CP.

· The UP failed to send packets to UP for processing.

· The physical link between the CP and UP fails.

Troubleshooting flow

Figure 19 shows the troubleshooting flowchart.

Figure 19 Flowchart for troubleshooting packet forwarding failure for the protocol channel

Solution

1. Check the physical link.

<Sysname> display interface ten-gigabitethernet 1/5/0

Ten-GigabitEthernet1/5/0

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

¡ Repeat the previous steps on the UP to check and repair the outgoing interface of the route to the CP.

¡ If other devices exist between the CP and UP, use the previous steps to check and repair the status of the connecting interfaces hop by hop.

¡ If the physical link between the CP and UP is normal and the issue persists, perform the following operations:

2. Check the VXLAN related settings for the protocol channel.

On the CP, execute the display current-configuration configuration up-manage command to view the detailed configuration for the protocol channel between the CP and UP.

<Sysname> display current-configuration configuration up-manage

up-manage id 1024

protocol-tunnel vxlan 10 source 2.2.2.2 destination 1.1.1.1

cu-agent

protocol-tunnel vxlan 10 source 1.1.1.1 destination 2.2.2.2

¡ On CTRL-VM, execute the cudetect cu tunnel-state up-id up-id command. If the Protocol Tunnel field displays NOK or NA, check and repair the UP-CP protocol channel as described in the step for checking the protocol channel configuration in “CUPS channel failure.”

¡ If the physical link between the CP and UP is normal and the issue persists, proceed to the following step:

3. Verify that the remote interface has been managed by the CP.

On UP, execute the display this command on the interface where users come online to check the cp-management configuration of the interface.

¡ If the configuration exists, it indicates that the interface has been managed remotely by the CP, and BRAS settings are issued normally.

¡ If the configuration does not exist, it indicates that the interface is not managed remotely by the CP. In this case, troubleshoot the management channel and control channel issues as described in "Management channel establishment failure" and "Control channel establishment failure."

¡ If the remote interface has been managed but the issue persists, proceed to the following step:

4. Check the protocol packet exchange between the CP and UP.

Perform repeated dial-up operations on the user end. Meanwhile, execute the display protocol-tunnel packet statistics command on the CP repeatedly at regular intervals (30 seconds as a best practice). View the packet statistics for the protocol channel and record the value of the Input packet statistics field each time displayed.

¡ If the value increases time by time, it indicates that the VXLAN protocol channel is normal.

¡ If the value does not increase, it indicates that the CP cannot receive protocol packets from the UP. Execute the display protocol-tunnel packet statistics command on the CP repeatedly at regular intervals (30 seconds as a best practice). Record the value of the Output packet statistics field each time.

<Sysname> display protocol-tunnel packet statistics

Input packet statistics:

Total: 7283

PPPoE PADI and PADO: 3

Other PPPoE: 0

DHCP DISCOVER and OFFER: 129

Other DHCP: 181

DHCPv6: 0

ND: 6970

L2TP: 0

ARP: 0

IPv4 data miss: 0

IPv6 data miss: 0

Ethernet: 0

IPv4: 0

IPv6: 0

Drop: 0

Output packet statistics:

Total: 1121

PPPoE PADI and PADO: 6

Other PPPoE: 0

DHCP DISCOVER and OFFER: 284

Other DHCP: 393

DHCPv6: 0

ND: 0

L2TP: 0

ARP: 0

IPv4 data miss: 417

IPv6 data miss: 21

Ethernet: 0

IPv4: 0

IPv6: 0

Drop: 0

If the value does not increase, execute the debugging ucm forward all command to enable UCM debugging. Collect the debugging information and proceed to the next step.

5. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Elastic scaling issues

VM manual scaling failure

Symptom

Failed to perform manual scaling with the VNFM-vBRAS.

Common causes

The following are the common causes of this type of issue:

· When manual scaling in is performed, a BRAS-VM has been associated with a UP.

· The link between vBRAS and VNFM-vBRAS fails.

· The configuration for the connection between vBRAS and VNFM-vBRAS is incorrect.

· The hardware resources on the server that holds the VMs are insufficient.

Troubleshooting flow

Figure 20 shows the troubleshooting flowchart.

Figure 20 Flowchart for troubleshooting VM manual scaling failure

Solution

1. Verify that the BRAS-VM is not associated with a UP.

For manual scaling out, go to step 2 regardless of whether the BRAS-VM has been associated with a UP.

For manual scaling in, execute the display bras-vm-up associated-info command on the CP to display the BRAS-VM and UP association information.

<Sysname> display bras-vm-up associated-info

Slot UP ID

129, 130 1024

¡ If the BRAS-VM is associated with a UP, execute the up-migrate to bras-vm command on the CP to migrate the UP to another BRAS-VM.

¡ If the BRAS-VM is not associated with a UP, go to step 2.

2. Check the link between vBRAS and VNFM-vBRAS.

If the CP outputs the following log information, it indicates that the link between vBRAS and VNFM-vBRAS has failed.

VMMGR/4/VMMGR_CREATE_FAIL: Failed to manually create VM 99 in group 67. Reason: Failed to connect to the vBRASSO server.

VMMGR/4/VMMGR_DELETE_FAIL: Failed to delete the manually created VM on slot 99 in group 67. Reason: Connection with the vBRASSO server timed out.

Execute the ping command on the CTRL-VM to test the connectivity to the IP address of the VNFM-VBRAS.

¡ If the VNFM-VBRAS IP cannot be pinged, examine all the links on the packet forwarding path and resolve the route issue.

¡ If the VNFM-VBRAS IP can be pinged, go to step 3.

3. Verify that the configuration for the connection between vBRAS and VNFM-vBRAS is correct.

The following conditions indicate that configuration errors exist for the connection between vBRAS and VNFM-vBRAS:

¡ The following log information is output on the CP:

VMMGR/4/VMMGR_CREATE_FAIL: Failed to manually create VM 99 in group 67. Reason: Failed to connect to the vBRASSO server.

VMMGR/4/VMMGR_DELETE_FAIL: Failed to delete the manually created VM on slot 99 in group 67. Reason: Connection with the vBRASSO server timed out.

¡ Execute the display vbras-cp stable state vnfm command on the CP. The command output shows that the communication status with VNFM is Not configured or Disconnected.

<Sysname> display vbras-cp stable state vnfm

------------------------------VNFM state------------------------------

VNFM communication state: Connected

Execute the display current-configuration command on the CP to display the VNFM-vBRAS configuration. Make sure the configuration in the vnfm address command is consistent with the actually used IP address, port number, username, and password for login to the VNFM-vBRAS and the actual mode (HTTP or HTTPS) for communication with the VNFM-vBRAS.

<Sysname> display current-configuration | include vnfm

vnfm address 192.168.73.33 user test password simple 123456789 http-method port 30000

¡ If the VNFM-vBRAS configuration is incorrect, execute the vnfm address command to edit the configuration.

¡ If the VNFM-vBRAS configuration is correct, go to step 4.

4. Verify that VMs have been deployed correctly.

The following log information output on the CP indicates VM deployment failures.

VMMGR/4/VMMGR_CREATE_FAIL: Failed to manually create VM 99 in group 67. Reason: The vBRASSO server failed to create the VM.

VMMGR/4/VMMGR_DELETE_FAIL: Failed to delete the manually created VM on slot 99 in group 67. Reason: The vBRASSO server failed to delete the VM.

¡ If a VM deployment failure exists, troubleshoot the issue as described in “VM deployment failure.”

¡ If VM deployment is correct, go to step 5.

5. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module name: HH3C-VNF-DEVICE-MIB

· hh3cVmCreateFail (1.3.6.1.4.1.25506.2.196.3.0.4)

· hh3cVmDeleteFail (1.3.6.1.4.1.25506.2.196.3.0.6)

Log messages

· VMMGR/4/VMMGR_CREATE_FAIL

· VMMGR/4/VMMGR_DELETE_FAIL

VM auto scaling failure

Symptom

Failed to perform auto scaling for VMs.

Common causes

The following are the common causes of this type of issue:

· Auto scaling for BRAS-VMs is not enabled.

· After the delay timer for auto scaling expires, UP migration conditions are not met.

· The link between vBRAS and VNFM-vBRAS fails.

· The configuration for the connection between vBRAS and VNFM-vBRAS is incorrect.

· The hardware resources on the server that holds the VMs are not enough.

Troubleshooting flow

Figure 21 shows the troubleshooting flowchart.

Figure 21 Flowchart for troubleshooting VM auto scaling failure

Solution

1. Identify whether the auto scaling is caused by UP count changes.

On the CP, execute the display bras-scale capacity command to display scaling capabilities for the current BRAS-VM.

<Sysname> display bras-scale capacity slot 129

Slot: 129, 130

Current UP count: 16

UP count threshold: 64

Current user count: 1000

Max user count: 2000000

User count lower threshold: 200000

User count alert threshold: 1600000

User count upper threshold: 1800000

Current delay time: 300s(will expand to 600s after 2 retry)

¡ If the Current UP count (number of UPs associated with the BRAS-VM) is greater than or equal to the UP count threshold (UP-count threshold for triggering auto scaling) or the Current UP count is 0, it indicates the auto scaling is caused by a CP count change. In this case, go to step 2.

¡ If the Current UP count is smaller than the UP count threshold and is not 0, it indicates the auto scaling is caused by a user count change. In this case, go to step 3.

2. Verify that BRAS-VM auto scaling is enabled.

On the CP, execute the display current-configuration command to display the enabling status of auto scaling for BRAS-VM.

<Sysname> display current-configuration | include bras-scale

bras-scale enable

¡ If BRAS-VM auto scaling is not enabled, enable it by using the bras-scale enable command in system view.

¡ If BRAS-VM auto scaling is enabled, go to step 3.

3. Identify whether BRAS-VM auto scaling has timed out.

The following log information on the CP indicates the BRAS-VM auto scaling timeout is reached.

VMMGR/4/VMMGR_CREATE_FAIL_FINAL: Failed to automatically create VM 99 in group 67 after the maximum number of retries reached.

VMMGR/4/VMMGR_DELETE_FAIL_FINAL: Failed to delete the automatically created VM on slot 99 in group 67 after the maximum number of retries reached.

On the CP, execute the display bras-scale capacity command to display the current delay time of auto scaling.

<Sysname> display bras-scale capacity slot 129

Slot: 129, 130

Current UP count: 16

UP count threshold: 64

Current user count: 1000

Max user count: 2000000

User count lower threshold: 200000

User count alert threshold: 1600000

User count upper threshold: 1800000

Current delay time: 300s(will expand to 600s after 2 retry)

¡ If the Current delay time is greater than the delay timer set by the bras-scale delay-time command, it indicates the auto scaling timeout is reached. Wait for the time indicated by Current delay time and then perform user online and offline operations.

¡ If the Current delay time is the same as the delay timer set by the bras-scale delay-time command same, it indicates the auto scaling timeout is not reached. Go to step 4.

4. Check the link between vBRAS and VNFM-vBRAS.

If the CP outputs the following log information, it indicates that the link between vBRAS and VNFM-vBRAS has failed.

VMMGR/4/VMMGR_CREATE_FAIL: Failed to manually create VM 99 in group 67. Reason: Failed to connect to the vBRASSO server.

VMMGR/4/VMMGR_DELETE_FAIL: Failed to delete the manually created VM on slot 99 in group 67. Reason: Connection with the vBRASSO server timed out.

Execute the ping command on the CTRL-VM to test the connectivity to the IP address of the VNFM-VBRAS.

¡ If the VNFM-VBRAS IP cannot be pinged, examine all the links on the packet forwarding path and resolve the route issue.

¡ If the VNFM-VBRAS IP can be pinged, go to step 5.

5. Verify that the configuration for the connection between vBRAS and VNFM-vBRAS is correct.

The following conditions indicate that configuration errors exist for the connection between vBRAS and VNFM-vBRAS:

¡ The following log information is output on the CP:

VMMGR/4/VMMGR_CREATE_FAIL: Failed to manually create VM 99 in group 67. Reason: Failed to connect to the vBRASSO server.

VMMGR/4/VMMGR_DELETE_FAIL: Failed to delete the manually created VM on slot 99 in group 67. Reason: Connection with the vBRASSO server timed out.

¡ Execute the display vbras-cp stable state vnfm command on the CP. The command output shows that the communication status with VNFM is Not configured or Disconnected.

<Sysname> display vbras-cp stable state vnfm

------------------------------VNFM state------------------------------

VNFM communication state: Connected

<Sysname> display current-configuration | include vnfm

vnfm address 192.168.73.33 user test password simple 123456789 http-method port 30000

¡ If the VNFM-vBRAS configuration is incorrect, execute the vnfm address command to edit the configuration.

¡ If the VNFM-vBRAS configuration is correct, go to step 6.

6. Verify that VMs have been deployed correctly.

If the following log information is output on the CP, it indicates that VM deployment failures exist.

VMMGR/4/VMMGR_CREATE_FAIL: Failed to manually create VM 99 in group 67. Reason: The vBRASSO server failed to create the VM.

VMMGR/4/VMMGR_DELETE_FAIL: Failed to delete the manually created VM on slot 99 in group 67. Reason: The vBRASSO server failed to delete the VM.

¡ If a VM deployment failure exists, troubleshoot the issue as described in “VM deployment failure.”

¡ If VM deployment is correct, go to step 7.

7. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

Module name: HH3C-VNF-DEVICE-MIB

· hh3cVmCreateFail (1.3.6.1.4.1.25506.2.196.3.0.4)

· hh3cVmDeleteFail (1.3.6.1.4.1.25506.2.196.3.0.6)

Log messages

· VMMGR/4/VMMGR_CREATE_FAIL

· VMMGR/4/VMMGR_CREATE_FAIL_FINAL

· VMMGR/4/VMMGR_DELETE_FAIL

· VMMGR/4/VMMGR_DELETE_FAIL_FINAL

UP allocation failure

Symptom

Execute the display bras-vm-up associated-info command on the CP to display the BRAS-VM and UP association information. The output shows an unmanaged UP exists, that is, the UP does not have BRAS-VM information.

<CP> display bras-vm-up associated-info

Slot UP ID

1024

129, 130 1025

Common causes

The following are the common causes of this type of issue:

· The number of UPs or users exceeds the BRAS-VM management capacity.

· The server resources are not enough for BRAS-VM auto scaling.

Troubleshooting flow

The primary troubleshooting procedure for this type of issue is as follows:

1. Identify whether the number of UPs exceeds the BRAS-VM management capacity.

2. Identify whether the number of users exceeds the BRAS-VM management capacity.

3. Identify whether the server resources are enough.

Figure 22 shows the troubleshooting flowchart.

Figure 22 Flowchart for troubleshooting UP allocation failure

Solution

1. Identify whether the number of UPs exceeds the BRAS-VM management capacity.

Execute the display bras-scale capacity command to display scaling capabilities for the current BRAS-VM.

<Sysname> display bras-scale capacity slot 129

Slot: 129, 130

Current UP count: 16

UP count threshold: 64

Current user count: 1000

Max user count: 2000000

User count lower threshold: 200000

User count alert threshold: 1600000

User count upper threshold: 1800000

Current delay time: 300s(will expand to 600s after 2 retry)

¡ If the Current UP count (number of UPs associated with the BRAS-VM) is greater than or equal to the UP count threshold (UP-count threshold for triggering auto scaling), it indicates the number of UPs has exceeded the BRAS-VM management capacity. Use the bras-scale capacity up-count-threshold command to change the UP-count threshold for auto scaling.

¡ If the Current UP count is smaller than the UP count threshold, it indicates the number of UPs has not exceeded the BRAS-VM management capacity. In this case, go to step 2.

2. Identify whether the number of user exceeds the BRAS-VM management capacity.

Execute the display bras-scale capacity command to display scaling capabilities for the current BRAS-VM.

<Sysname> display bras-scale capacity slot 129

Slot: 129, 130

Current UP count: 16

UP count threshold: 64

Current user count: 1000

Max user count: 2000000

User count lower threshold: 200000

User count alert threshold: 1600000

User count upper threshold: 1800000

Current delay time: 300s(will expand to 600s after 2 retry)

¡ If the Current user count (number of users on UPs managed by the BRAS-VM) is greater than or equal to the User count upper threshold (user-count threshold for triggering auto scaling), it indicates the number of users has exceeded the BRAS-VM management capacity. Use the bras-scale capacity user-count-threshold command to change the user-count threshold for auto scaling.

¡ If the Current user count is smaller than the User count upper threshold, it indicates the number of users has not exceeded the BRAS-VM management capacity. In this case, go to step 3.

3. Identify whether the server resources are enough.

After an auto scaling failure, log in to the CAS management interface of the server host to view VM resource usage information.

¡ If the server resources are not enough, scale out server resources.

¡ If the server resources are enough, go to step 4.

4. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

None.

Log messages

· UPLB_SCALE_EXPAND_FAILED

· UPLB_SCALE_SHRINK_FAILED

CP disaster recovery issues

Symptom

· In hot backup mode, after a user comes online from the master CP, user information cannot be backed up to the backup CP.

· The CPs cannot negotiate the master and backup roles correctly. Two master CPs or two backup CPs exist.

· A user cannot come online from the new master CP after a master-backup switchover.

Common causes

The following are the common causes of this type of issue:

· Routing failure.

· The heartbeat channel between the master and backup CPs was not established.

· The data backup channels between the master and backup CPs were not established.

· Incorrect source interface configuration for outgoing RADIUS packets.

· Inconsistent configuration on the master and backup CPs.

Troubleshooting flow

Figure 23 shows the troubleshooting flowchart:

Figure 23 Flowchart for troubleshooting CP disaster recovery failure

Solution

1. Identify whether the master and backup CPs can reach each other at Layer 3.

On one CP, ping the other CP. If the ping succeeds, proceed to the next step. If the ping fails, resolve the routing issue.

2. Identify whether the master/backup CPs and the UP can reach each other at Layer 3.

On the UP, ping the master and backup CPs respectively. If the ping succeeds, proceed to the next step. If the ping fails, resolve the routing issue.

3. Identify whether the master CP and the servers (such as AAA server) can reach each other at Layer 3.

On the master CP, ping the AAA server. If the ping succeeds, proceed to the next step. If the ping fails, resolve the routing issue.

4. Identify whether the master and backup CPs have consistent configuration.

On the master and backup CPs, execute the display current-configuration command. Compare the configurations on the master and backup CPs, such as the IP address pool configuration and the source interface configuration for outgoing RADIUS packets. If the configuration is consistent, proceed to the next step. If the configuration is inconsistent, edit the configuration to ensure consistency.

5. Verify that the CPDR channel between the master and backup CPs is normal.

Perform the following operations to check the CPDR channel between the master and backup CPs:

¡ Execute the display cp disaster-recovery data-tunnel command to display data backup channel information. If no data backup channels are established, check the data channel configuration, network configuration, and link connection status.

¡ Execute the display cp disaster-recovery heartbeat-tunnel command to display TCP heartbeat channel information. If no heartbeat channel is established, check the heartbeat channel configuration, network configuration, and link connection status.

¡ Execute the display cp disaster-recovery protect-tunnel statistics command to display packet statistics for the protection channel. If packet statistics are abnormal, check the related feature configuration, network configuration, and link connection status.

¡ Execute the display cp disaster-recovery group command to display CPDR group configuration and running information. If the CUSP channel is not set up correctly, proceed to the next step.

6. Identify whether the CU connections are set up correctly.

On the CPs, execute the display cusp controller command to display the connection information of the CUSP controller.

On UPs, execute the display cusp agent command to display the connection information of the CUSP agent.

If the CUSP channel connections are abnormal, check the CUSP configuration and resolve the issue as described in the CUSP connection troubleshooting guide.

7. Identify whether the device state is stable.

On CPs, execute the display vbras-cp stable state command to identify whether the CP and UP separation system is in stable state.

8. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file, log messages, and alarm messages.

UP backup issues

Master/backup interface failure or master/backup switchover

Symptom

The master and backup interfaces in a UP backup profile are operating incorrectly. The output from the display up-backup-profile command executed on the CP shows the following errors:

· The state value for the master interface is not master(normal).

· The state value for the backup interface is not backup(normal).

<Sysname> display up-backup-profile

...

Interface group 1:

Master: Remote-XGE1024/1/0/1, state=backup(normal), VRID=1

Backup: Remote-XGE1025/1/0/1, state=master(normal)

...

Common causes

The following are the common causes of this type of issue:

· The physical links of the master and backup interfaces on the UPs go down.

· CUSP channel failure occurs between the CP and the UPs that host the master and backup interfaces.

· The track entries on the UPs are in an abnormal state.

· Switchover back to the original master interface upon failure recovery is disabled in the UP backup profile on the CP.

Troubleshooting flow

1. Verify that the physical links of the master and backup interfaces on the UPs are operating correctly.

2. Identify the cause of the issue from the backup profile information.

3. Verify that the CUSP channels between the CP and the UPs are operating correctly.

4. Verify that the track entries on the UPs are in a normal state.

5. Verify that switchover back to the original master interface upon failure recovery is enabled in the UP backup profile on the CP.

Figure 24 shows the troubleshooting flowchart.

Figure 24 Flowchart for troubleshooting master/backup interface failure or switchover

Solution

1. Verify that the physical links of the master and backup interfaces on the UPs are operating correctly.

Execute the display interface command on the UPs to view interface information, for example:

<Sysname> display interface ten-gigabitethernet 3/1/1

Ten-GigabitEthernet3/1/1

Interface index: 386

Current state: Administratively DOWN

Line protocol state: DOWN

...

Perform the following steps depending on the state of the interface:

a. If the Current state field displays Administratively DOWN, execute the undo shutdown command on the interface to bring up the interface.

b. If the Current state field displays DOWN, check the interface cabling for physical connection issues.

c. Repeat the previous steps on the UPs to verify that the outgoing interfaces to the CP are operating correctly.

2. Execute the display up-backup-profile command on the CP to view UP backup profile information.

¡ If the Reason field displays CUSP down, the CUSP channel between the CP and the UP that hosts the master interface is operating incorrectly. Go to step 3.

¡ If the Reason field displays Track negative, the network interface monitored by the UP through Track goes down. Go to step 4.

¡ If the Failure recovery field displays Disabled, switchover back to the original master interface upon failure recovery is disabled. Go to step 5.

3. Verify that the CUSP channels between the CP and the UPs that host the master and backup interfaces are operating correctly.

Execute the display cusp controller command on the CP to view information about connections to the UPs on the CUSP controller.

¡ If the Control tunnel state field displays Inactive, troubleshoot the issue as described in "Control channel establishment failure."

¡ If the Control tunnel state field displays Active, the CUSP channel is operating correctly.

4. Identify the track monitoring states on the UPs.

Execute the display this command in UP backup profile view on the CP to identify whether the up-id up-id network-state track uplink-group group-name command is executed.

¡ If not, identify the track entry state on the UP. If the state is Positive, go to next step.

¡ If yes, verify that the user-plane switchover track track-id uplink-group group-name command is executed on the UP that hosts the master interface. Make sure the CP and UP are configured with the same UP uplink network resource group. Then, execute the display track track-id command to view the state of the corresponding track entry associated with the UP. If the State field displays Negative for the track entry, the tracked object is operating incorrectly. Troubleshoot tracked object exceptions according to the corresponding information.

<Sysname> display track all

Track ID: 2

State: Negative

Duration: 0 days 0 hours 0 minutes 32 seconds

Tracked object type: BFD

Notification delay: Positive 20, Negative 30 (in seconds)

Tracked object:

BFD session mode: Echo

Outgoing interface: Ten-GigabitEthernet3/1/1

...

5. Verify that switchover back to the original master interface upon failure recovery on the CP is correctly configured.

Execute the display this command in UP backup profile view on the CP to identify whether switchover back to the original master interface upon failure recovery is enabled:

¡ If not, execute the failure-recovery-switch enable command to enable switchover back to the original master UP or interface upon failure recovery.

¡ If yes, verify that an appropriate switchover delay is set, for example, 30 seconds. A long delay might result in delayed switchovers. A short delay might cause frequent master/backup switchovers.

6. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file and alarm messages.

¡ Execute the display system internal up-backup log event command to obtain information about UP backup event log messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

· UPBAK/6/UPBAK_IF_STATE_CHANGE

· UPBAK/6/UPBAK_IF_STATE_SWITCH

Long master/backup interface switchover

Symptom

A long master/backup interface switchover might cause a long traffic interruption due to the following errors:

· The backup interface fails to take over in time when the master interface fails.

· Switchover back to the original master interface upon failure recovery is not in time.

Common causes

The following are the common causes of this type of issue:

· A long switchover delay is set for switchover back to the original master interface upon failure recovery.

· A long switchover delay upon CUSP channel failure is set.

· A long switchover delay upon CUSP channel failure recovery is set.

· Service modules are slow in switchover processing.

Troubleshooting flow

1. Verify that the switchover delay is appropriate for switchover back to the original master interface upon failure recovery.

2. Verify that the switchover delay upon CUSP channel failure is appropriate.

3. Verify that the switchover delay upon CUSP channel failure recovery is appropriate.

4. Verify that the service modules are operating correctly in processing switchovers.

Figure 25 shows the troubleshooting flowchart.

Figure 25 Flowchart for troubleshooting long master/backup interface switchover

Solution

1. Execute the display up-backup-profile command on the CP to identify whether the delay is appropriate, for example:

<Sysname> display up-backup-profile 1

Profile ID: 1

Backup mode: Hot standby

Failure recovery: Enabled Delay time: 1800 seconds

CUSP tunnel down switchover Delay time: 1800 seconds

CUSP tunnel up switchover Delay time: 60000 milliseconds

Route advertise: Disabled

Interface backup mode: Inherit-main

Interface group 1:

Master: Remote-XGE2009/1/3/0, state=backup(normal), VRID=2

Backup: Remote-XGE2000/1/3/0, state=master(normal)

Switchback state: Waiting(remaining time: 1797 seconds)

¡ If the Failure recovery field displays Enabled, switchover back to the original master interface upon failure recovery is enabled. The value range for the switchover delay is 0 to 1800, in seconds, and the default is 30 seconds. If the delay is much longer than 30 seconds, go to step 2.

¡ If the Delay time field for CUSP tunnel down switchover displays the switchover delay upon CUSP channel failure, in the range of 0 to 1800 seconds. By default, the CP notifies the UP to perform a master/backup UP or interface switchover 50 milliseconds after CUSP channel failure occurs between the CP and a UP. If the delay is much longer than 50 milliseconds, go to step 3.

¡ If the Delay time field for CUSP tunnel up switchover displays the switchover delay upon CUSP channel failure recovery, in the range of 0 to 60000 milliseconds. By default, the CP notifies the UP to perform a master/backup UP or interface switchover 3 seconds after a CUSP channel failure recovery. If the delay is much longer than 3 seconds, go to step 4.

2. If the switchover delay upon failure recovery is long, modify the delay.

On the CP, execute the failure-recovery-switch enable [ delay delay-time ] command in UP backup profile view or CGN-UP backup profile view to specify an appropriate delay.

3. If the switchover delay upon CUSP channel failure is long, modify the delay.

On the CP, execute the control-tunnel-down switchover [ delay sec-delay-time | msec-delay msec-delay-time ] command in UP backup profile view or CGN-UP backup profile view to specify an appropriate delay.

4. If the switchover delay upon CUSP channel failure recovery is long, modify the delay.

On the CP, execute the control-tunnel-up switchover msec-delay delay-time command in UP backup profile view or CGN-UP backup profile view to specify an appropriate delay.

5. Wait another 60 seconds after the switchover delay expires.

6. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file and alarm messages.

¡ Execute the display system internal up-backup log event to obtain information about UP backup event log messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Two master interfaces on UPs

Symptom

When the CP notifies the UPs to switch the master interface to backup and the backup interface to master, the master interface fails to switch to backup. As a result, two master interfaces exist. In this case, user access devices connected to the UPs repeatedly refresh forwarding interface information, resulting in forwarding entry flapping, which causes packet loss.

When you execute display system internal up interface-backup on the UPs that host the master and backup interfaces, the output shows that the states of the interfaces are Master.

<Sysname> system-view

[Sysname] probe

[Sysname-probe] display system internal up interface-backup

Interface: Ten-GigabitEthernet3/1/4

IfIndex: 65

State: Master

Backup mode: Hot standby

Interface backup mode: Inherit-main

Resource ID: 0x20001

Virtual MAC: 0000-5e00-0101

Switchover upon ctrl tunnel down: Enabled

Switchover delay: 0

Common causes

The following are the common causes of this type of issue:

· CUSP channel failure occurs between the CP and the UP that hosts the master interface and the master/backup interface switchover is disabled.

· The UCM service module fails to notify the UP backup module to switch the master interface to backup.

Troubleshooting flow

1. Locate the cause for the master/backup switchover and identify whether the master/backup interface switchover is disabled on the UPs.

2. Recover the CUSP channels between the CP and UPs.

Figure 26 shows the troubleshooting flowchart.

Figure 26 Flowchart for troubleshooting two master interfaces on UPs

Solution

1. Execute the display up-backup-profile profile-id switch-history command on the CP to locate the cause of the most recent switchover, for example:

<Sysname> display up-backup-profile 1 switch-history

Reason Interface State Time

CUSP down Remote-XGE2009/1/3/0 Switchover to backup 2021-08-30 04:28:39

¡ If the Reason field displays CUSP down, the switchover is caused by a CUSP channel failure. Go to step 2.

¡ If the Reason field displays CUSP down, the service modules might be faulty. Go to step 3.

2. Perform the following operations:

¡ Verify UP settings. Execute the display current-configuration command to identify whether master/backup interface switchover is enabled on the UPs. To enable master/backup interface switchover on a UP, execute the user-plane control-tunnel-down switchover track command.

¡ Execute the display cusp controller command on the CP to view information about connections to the UPs on the CUSP controller.

- If the Control tunnel state field displays Inactive, troubleshoot the issue as described in "Control channel establishment failure."

- If the Control tunnel state field displays Active, the CUSP channel is operating correctly.

3. If the issue persists, collect the following information and contact Technical Support:

¡ Results of each step.

¡ The configuration file and alarm messages.

¡ Execute the display system internal up-backup log event command to obtain information about UP backup event log messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Two backup interfaces on UPs

Symptom

Two backup interfaces exist on UPs when the following conditions are met:

· The CP fails to notify the UPs to perform a master/backup switchover when CUSP channel failure occurs between the CP and the UPs that host the master and backup interfaces.

· The master and backup interfaces are faulty.

In this case, users cannot come online because the master and backup UPs fail to process user traffic.

When you execute display system internal up interface-backup on the UPs that host the master and backup interfaces, the output shows that the states of the interfaces are Backup.

<Sysname> system-view

[Sysname] probe

[Sysname-probe]display system internal up interface-backup

Interface: Ten-GigabitEthernet3/1/4

IfIndex: 65

State: Backup

Backup mode: Hot standby

Interface backup mode: Inherit-main

Resource ID: 0x20001

Virtual MAC: 0000-5e00-0101

Switchover upon ctrl tunnel down: Enabled

Switchover delay: 0

Common causes

The following are the common causes of this type of issue:

· The following conditions are all met:

¡ The master and backup interfaces are faulty.

¡ CUSP channel failure occurs between the CP and UPs.

¡ The master/backup interface switchover is disabled on the UPs.

· The UCM service module fails to notify the UP backup module to switch the backup interface to master.

Troubleshooting flow

1. Verify that the physical links of the master and backup interfaces on the UPs are operating correctly.

2. Verify that the CUSP channels between the CP and the UPs are operating correctly.

3. Collect information about UP backup event log messages.

Figure 27 shows the troubleshooting flowchart.

Figure 27 Flowchart for troubleshooting two backup interface on UPs

Solution

1. Verify that the physical links of the master and backup interfaces are operating correctly.

Execute the display interface command on the UPs, for example: