Troubleshooting hardware

Card issues

Abnormal card status

Symptom

·     A card is abnormal. (For example, the card status displays Absent or Abnormal after you execute the display device command.)

·     A card fails to boot, or it reboots unexpectedly or repeatedly.

Common causes

The following are the common causes for this type of issue:

·     The card is not securely installed.

·     The card is damaged.

·     Lighting of LEDs on the card panel is abnormal.

·     A power supply has failed.

·     The power supply output power is insufficient.

·     The host software version does not support the card.

·     The MPU is not operating correctly.

Troubleshooting flow

Figure 1 shows the troubleshooting flowchart.

Figure 1 Flowchart for troubleshooting the issue of abnormal card status

‌

Solution

Card in Absent status

1.     Identify whether the card is securely installed. Examine for gaps between the card and the chassis. You can also reinstall the card. Before reinstallation, make sure the connector of the card is not distorted or dirty.

2.     Move the card to another slot, or move a normal card from another slot to the slot where the card is installed. This operation helps you identify whether the card is faulty.

3.     Identify whether the LEDs on the card panel are lit.

4.     Identify whether the power supply output power is insufficient. For example, add power supplies and identify whether the card status restores to normal.

5.     Identify whether the host software version supports the card.

a.     Execute the display version command to view the software version of the host.

b.     Contact Technical Support to identify whether the current software version of the host supports the card.

c.     If the current software version does not support the card, upgrade it to a compatible version. Before version upgrade, make sure the new version is compatible with other cards.

6.     If the card is an MPU, connect a console cable to it and then use a fine tool (such as a pen tip) to press the system reset button (RESET) on the card. You can also reboot the card by using the reboot slot slotid force command. Identify whether the startup information displayed on the configuration terminal and the status LEDs on the card restore to normal. (It is abnormal that the configuration terminal displays nothing or garbled text.) Under normal conditions, the configuration terminal displays startup information similar to the following:

System is Starting....

Press Ctrl+D to access BASIC-BOOTWARE MENU

Press Ctrl+T to access BOOTWARE DIAG-TEST MENU

Booting Normal Extend BootWare

 

****************************************************************************

*                                                                          *

*                         BootWare, Version 1.35                           *

*                                                                          *

****************************************************************************

 

Compiled Date         : Dec  9 2021

Memory Type           : DDR4 SDRAM

Memory Size           : 16384MB

Memory Speed          : 2133MHz

flash Size            : 7296MB

CPLD 1 Version        : 4.0

CPLD 2 Version        : 1.0

CPLD 3 Version        : 1.0

PCB 1 Version         : Ver.A

PCB 2 Version         : Ver.A

 

 

BootWare Validating...

Press Ctrl+B to access EXTENDED-BOOTWARE MENU...

Loading the main image files...

Loading file flash:/SYSTEM.bin..............................................

............................................................................

............................................................................

....................................Done.

Loading file flash:/BOOT.bin................................................

............................................................................

............................................................................

............................................................................

................Done.

 

Image file flash:/BOOT.bin is self-decompressing............................

............................................................................

.............................................Done.

System image is starting...

 

Cryptographic algorithms tests passed.

 

Line aux0 is available.

 

Press ENTER to get started.

7.     If the card is a switching fabric module with a console port, connect a console cable to it. Then, execute the reboot slot slotid force command or reinstall the card to reboot the card. Identify whether the startup information displayed on the configuration terminal and the status LEDs on the card restore to normal.

8.     If the card is a interface module, first ensure that the MPU is operating correctly and that the subcard connectors are not deformed or dirty.

9.     If you confirm that the card is faulty, replace it. Collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Card in Abnormal status

10.     ‍Check the system power consumption. If the system power consumption is insufficient, the card will enter Abnormal status.

11.     Wait about 10 minutes to identify whether the card remains in Abnormal status or is in Normal status and then reboots again. If the card is in Normal status and automatically reboots, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

12.     If the card is an MPU, connect a console cable to the card, and then identify whether the configuration terminal displays information about correct card startup or any startup issues. If the MPU has a memory read/write test failure during startup and continuously reboots, identify whether the memory module is securely installed.

readed value is 55555555 , expected value is aaaaaaaa

DRAM test fails at: 080ffff8

DRAM test fails at: 080ffff8

Fatal error! Please reboot the board.

13.     Move the card to another slot to identify whether the slot is faulty.

14.     If you confirm that the card is faulty, replace it. Collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Card reboot anomaly

A card reboot refers to the situation where the status of the card is normal after it reboots.

15.     ‍Determine whether a user rebooted the card by using the reboot command or by powering off and then powering on the card during the period.

16.     You can use the display version command to obtain the reason for the most recent reboot of the card. For example, Last reboot reason indicates that the reason for the most recent reboot of the card was that the device was powered on.

<Sysname> display version

H3C Comware Software, Version 7.1.075, Release 7751P01

Copyright (c) 2004-2017 New H3C Technologies Co. Ltd. All rights reserved.

H3C xxx uptime is 0 weeks, 0 days, 4 hours, 24 minutes

Last reboot reason : Cold reboot……

17.     If all cards reboot simultaneously, verify the following information:

¡     The power supplies are operating correctly.

¡     The external power source does not have a power outage.

¡     The power cables are connected securely.

18.     If you cannot confirm the above information, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

MPU startup failure

Symptom

The original MPU or the standby MPU newly installed on the device cannot start up.

Common causes

The following are the common causes for this type of issue:

·     The MPU cannot be powered up due to hardware failure.

·     The basic section of BootWare for the MPU is damaged.

·     The BootWare cannot operate due to memory or CPU hardware failure.

·     The app software version is lost or does not match the hardware, or the app software version verification has failed.

·     The model of the standby MPU is different from that of the original MPU.

·     The software versions of the standby MPU and the original MPU are different.

Troubleshooting flow

Figure 2 shows the flowchart for troubleshooting the issue of original MPU startup failure.

Figure 2 Flowchart for troubleshooting the issue of original MPU startup failure

 

Figure 3 shows the flowchart for troubleshooting the issue of startup failure of the standby MPU newly installed on the device.

Figure 3 Flowchart for troubleshooting the issue of startup failure of the standby MPU newly installed on the device

 

Solution

To troubleshoot the issue of original MPU startup failure:

1.     Identify whether the running status LED (RUN) on the MPU is on.

This serves as an important indicator of whether the system can boot because the RUN LED will be steady on after the basic section of BootWare starts up.

¡     Situation 1: The LED flashes slowly.

If the LED flashes green slowly at 1 Hz after you power on the device, the basic section starts up normally. Proceed to step 2.

¡     Situation 2: The LED is off.

If the LED is off, the device cannot be powered on or the basic section of BootWare is damaged.

First, identify whether the device is powered on. Identify whether the internal MPU has a LED flashes green or is steady on by observing from the front of the MPU air inlet vents. You can also remove the MPU after a period of time and examine the processor's heat sink for warmth. If the device is not powered on, check the power source and power supplies. Hardware faults can also prevent the MPU from being powered on.

If the device is powered on normally, the basic section of BootWare is damaged and must be returned to R&D for handling.

 

	NOTE: In this situation, the LED has never been on after poweron. This situation does not include the case where the LED flashes for more than 5 seconds and then turns off.

 

2.     Identify whether BootWare runs successfully.

¡     Situation 1: The basic section runs successfully.

Identify whether the following information exists. If yes, the basic section has run successfully. Proceed to step 3.

System is Starting....

Press Ctrl+D to access BASIC-BOOTWARE MENU

Press Ctrl+T to access BOOTWARE DIAG-TEST MENU

Booting Normal Extend Bootware

 

****************************************************************************

*                                                                          *

*                         BootWare, Version 0.22                           *

*                                                                          *

****************************************************************************

Copyright (c) 2004-2019 New H3C Technologies Co., Ltd.

 

Compiled Date         : Mar 22 2019

Memory Type           : DDR4 SDRAM

Memory Size           : 8192MB

Memory Speed          : 1866MHz

flash Size            : 3728MB

CPLD Version          : 12.0

PCB Version           : Ver.A

 

 

BootWare Validating...

¡     Situation 2: No output.

The memory or processor might be faulty. Proceed to step 3.

3.     Identify whether apps can be loaded correctly.

¡     Situation 1: The app files can be loaded and decompressed successfully.

The following information indicates that the app files have been loaded and decompressed successfully. Proceed to step 4.

****************************************************************************

*                                                                          *

*                         BootWare, Version 0.22                           *

*                                                                          *

****************************************************************************

Copyright (c) 2004-2019 New H3C Technologies Co., Ltd.

 

Compiled Date         : Mar 22 2019

Memory Type           : DDR4 SDRAM

Memory Size           : 8192MB

Memory Speed          : 1866MHz

flash Size            : 3728MB

CPLD Version          : 12.0

PCB Version           : Ver.A

 

 

BootWare Validating...

Press Ctrl+B to access EXTENDED-BOOTWARE MENU...

Loading the main image files...

Loading file flash:/ra5300rsu3xx-cmw710-system-e0801.bin...

.................

............................................................................

.......Done.

Loading file flash:/ra5300rsu3xx-cmw710-boot-e0801.bin.....................

Done.

 

Image file flash:/ra5300rsu3xx-cmw710-boot-e0801.bin is

self-decompressing...................Done.

¡     Situation 2:  An app does not exist.

The following information indicates that an app file does not exist. The app file must be downloaded again.

****************************************************************************

*                                                                          *

*                         BootWare, Version 0.22                           *

*                                                                          *

****************************************************************************

Copyright (c) 2004-2019 New H3C Technologies Co., Ltd.

 

Compiled Date         : Mar 22 2019

Memory Type           : DDR4 SDRAM

Memory Size           : 8192MB

Memory Speed          : 1866MHz

flash Size            : 3728MB

CPLD Version          : 12.0

PCB Version           : Ver.A

 

 

BootWare Validating...

Application program does not exist.

Please input BootWare password:

¡     Situation 3: An app file has a CRC error.

The following information indicates that an obtained app file has a verification error. Please download the file to flash memory again.

****************************************************************************

*                                                                          *

*                         BootWare, Version 0.22                           *

*                                                                          *

****************************************************************************

Copyright (c) 2004-2019 New H3C Technologies Co., Ltd.

 

Compiled Date         : Mar 22 2019

Memory Type           : DDR4 SDRAM

Memory Size           : 8192MB

Memory Speed          : 1866MHz

flash Size            : 3728MB

CPLD Version          : 12.0

PCB Version           : Ver.A

 

 

BootWare Validating...

Press Ctrl+B to enter extended boot menu...

Loading file flash:/SYSTEM-.bin..................

............................................................................

............................................................................

............................................................................

Something wrong with the file.

4.     Check the app startup process.

¡     Situation 1: Without the system image file, the system starts up and enters the boot interface.

Loading the main image files...

Loading file flash:/BOOT.bin....................

...................................Done.

<boot>

In this case, you must download the software version again.

¡     Situation 2: The System image is starting... message is displayed and the system gets stuck.

¡     Situation 3: The System image is starting... message is displayed, but the system fails to enter the CLI and reboots repeatedly.

¡     Situation 4: The Press ENTER to get started message is displayed, but you cannot access the CLI.

¡     Situation 5: You can access the CLI, but the system automatically reboots after a while.

In these situations, a hardware failure or software version issue might occur. Please contact Technical Support.

To troubleshoot the issue of startup failure of the standby MPU newly installed on the device.

5.     Identify whether the model of the newly installed MPU is the same as that of the original MPU.

The two MPUs on the same device must be the same model. If their models are different, install an MPU of the same model as the original one.

6.     Collect diagnostics information.

Check the operating status of the active MPU, collect diagnostics information, and contact Technical Support.

7.     Contact Technical Support.

If the issue persists, contact Technical Support.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

An MPU restarts during use and fails to start up

Symptom

An MPU restarts during use and fails to start up.

Common causes

The following are the common causes for this type of issue:

·     The startup file is damaged.

·     The MPU memory is damaged.

·     The card is not fully inserted or is damaged, causing BootWare to run abnormally.

Troubleshooting flow

Figure 4 shows the troubleshooting flowchart.

Figure 4 Flowchart for troubleshooting the issue that the MPU restarts during use and fails to start up

 

Solution

1.     Identify whether the startup file on the MPU is normal.

Log in to the faulty MPU through the console port. Restart the device. If BootWare prompts a CRC error or the startup file cannot be found, reload the startup file and identify whether the size of the file in flash memory is the same as that on the server. If the flash memory does not have the file or the size of the file in flash memory differs from that on the server, reload the startup file. Then, configure the reloaded file as the current startup file. BootWare can automatically configure this file as the current startup file during the loading process.

2.     Examine the MPU memory.

If the loaded file size is correct and the file is correctly set as the current startup file, reboot the card and immediately press CTRL+T to examine the memory. If a memory error is prompted, please replace the card.

3.     Identify whether BootWare still prompts an error.

If the memory is normal but BootWare still prompts an error during startup, identify the faulty component based on the prompt. Identify whether the card is securely inserted. Replace the card if it is securely inserted.

4.     Contact Technical Support.

If the issue persists, contact Technical Support.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Active/standby MPU switchover failure

Symptom

·     When you use the reboot command to reboot the active MPU, the standby MPU is also rebooted.

·     Active/standby MPU switchover is abnormal.

Common causes

The following are the common causes for this type of issue:

·     If the original standby MPU has not completed startup, it passively becomes the active MPU because of the active MPU reboot.

·     The standby MPU does not receive any packets from the active MPU and switches to the active MPU.

·     The active MPU reboots due to its own anomalies.

·     The versions of the standby MPU and the active MPU are different.

Troubleshooting flow

Figure 5 shows the flowchart for troubleshooting the issue that the standby MPU is also rebooted when you use the reboot command to reboot the active MPU.

Figure 5 Flowchart for troubleshooting the issue that the standby MPU is also rebooted when you use the reboot command to reboot the active MPU

 

Solution

To troubleshoot the issue that the standby MPU is also rebooted when you use the reboot command to reboot the active MPU:

1.     After the original active MPU starts up, use the ftp or tftp command to upload the up-to-date log file in the log file directory on the storage media to the file server.

2.     Search the log file for the reboot log message, for example, Command is reboot slot 0.

3.     Search the log file for the most recent system restart log message, for example, SYSLOG_RESTART: System restarted.

4.     Search the log messages between the two log messages for a log message like Batch backup of standby board in slot 1 has finished.

¡     If no log message like the specified message is found, the original standby MPU was starting up when you executed the reboot command. This is normal and requires no action. Next time you want to use the reboot slot command to reboot the active MPU, make sure the standby MPU has completed batch backup (a log message like Batch backup of standby board in slot 1 has finished already exists).

¡     If a log message like the specified message is found, contact Technical Support.

To troubleshoot an active/standby MPU switchover failure:

5.     Use the display system stable state command to collect information about the active and standby MPU status.

<H3C> display system stable state                                                

System state     : Stable                                                        

Redundancy state : Stable                                                        

  Slot    CPU    Role       State                                                

  0       0      Active     Stable                                               

  1       0      Standby    Stable

Verify the following information:

¡     The roles of the two MPUs are Active and Standby.

¡     Both the active and standby MPUs are in Stable status.

6.     Use the display boot-loader command to collect information about the versions of the active and standby MPUs. Identify whether the versions of the active and standby MPUs are the same.

Fault diagnostics commands

The commands required for fault diagnostics are shown in the following table.

You can execute the following commands to enter probe view:

<Sysname> system

[Sysname] probe

[Sysname-probe]

 

Command	View	Description
display kernel exception number slot slot-num	Any view	Display exception information.
display system stable state	Any view	Display the current status of the active and standby MPUs.
display boot-loader	Any view	Display information about the versions of the active and standby MPUs.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

Interface module startup failure

Symptom

An interface module cannot start up.

Common causes

The following are the common causes for this type of issue:

·     Power supply anomaly.

·     The software version does not support the interface module.

·     The interface module is not securely installed.

·     Interface module hardware failure.

·     Chassis slot hardware failure.

Troubleshooting flow

Figure 6 shows the troubleshooting flowchart.

Figure 6 Flowchart for troubleshooting interface module startup failure

‌

Solution

1.     Identify whether the interface module is powered on.

Check the RUN LED on the interface module. If the LED is off, the interface module might not be powered on. Perform the following tasks:

a.     Examine the power status LEDs to determine whether the power supplies are operating normally. If a LED indicates an error, see the abnormal power supply status troubleshooting procedure for power supply troubleshooting.

b.     Calculate the system power consumption. Identify whether the remaining power of power supplies is sufficient. If the remaining power is insufficient, increase power supplies.

c.     If the interface module is powered on, proceed to step 3.

2.     Identify whether the device software version supports the interface module.

In any view, execute the display version command to obtain the device's software version. Then, identify whether the current software version supports the interface module. If not, please upgrade the version to one that supports the interface module. Before version upgrade, make sure the new version is compatible with other cards.

3.     Reinstall the interface module.

Remove the interface module, verify the connector, and then reinsert it into the device. Make sure the interface module is installed securely.

4.     Install the interface module in another slot to test if it can start up.

¡     If the interface module cannot start up, it might be faulty. Replace it with a new one.

¡     If the interface module can start up, install another interface module that can start up normally in the original faulty slot. If the interface module cannot start up, the chassis slot might be faulty.

5.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

N/A

An interface module restarts during use and fails to start up

Symptom

An interface module restarts during use and fails to start up.

Common causes

The following are the common causes for this type of issue:

·     Power supply anomaly.

·     The startup file on the MPU is abnormal.

·     Interface module hardware failure.

·     Chassis slot hardware failure.

Troubleshooting flow

Figure 7 shows the troubleshooting flowchart.

Figure 7 Flowchart for troubleshooting the issue that the interface module restarts during use and fails to start up

‌

Solution

1.     Identify whether the power supplies are operating normally.

Verify that the power status LEDs indicate normal status and the power meets the normal operation requirements of cards. If a power supply malfunctions, see the abnormal power supply status troubleshooting procedure for power supply troubleshooting.

2.     Identify whether the startup file on the MPU is normal.

Execute the display boot-loader command in any view to check the next-startup software image used by the card. Execute the dir command in user view to identify whether the startup software image exists. If it does not exist or is damaged, retrieve the startup software image again or set another software image as the next-startup software image.

3.     Insert an interface module that can operate correctly into the slot where the interface module cannot start up.

If the startup file loaded by the interface module is normal and conditions permit, insert an interface module that can operate correctly into the slot where the interface module cannot start up.

¡     If the interface module can start up, the MPU and backplane are normal. Proceed to step 4.

¡     If the interface module cannot start up, replace the MPU.

4.     Identify whether load records exist.

Execute the display logbuffer command in any view to identify whether the log buffer on the device has load records for the card.

<Sysname> display logbuffer

%Jan 12 19:13:49:513 2022 H3C DEV/4/BOARD_LOADING: -MDC=1; Board in slot 4 is loading software images.

%Jan 12 19:14:01:718 2022 H3C DEV/5/LOAD_FINISHED: -MDC=1; Board in slot 4 has finished loading software images.

¡     If the log buffer has load records for the card, move the interface module to another slot and identify whether it can start up normally.

¡     If the log buffer does not have load records for the card, proceed to step 5.

5.     If the issue persists, collect the following information and contact Technical Support:

¡     Results of each step.

¡     The configuration file, log messages, and alarm messages.

Related alarm and log messages

Alarm messages

N/A

Log messages

·     DEV/4/BOARD_LOADING

·     DEV/5/LOAD_FINISHED