1.1  动态NAT转换故障(以动态nat outbound为例)

1.1.1  故障描述

图1-1 动态NAT转换组网图

 

2. 组网需求

Host 1访问Host 2，在Device上对Host 1的地址进行NAT转换，转换地址池为：4.4.4.25到4.4.4.30。Device上有2块防火墙业务板。

3. Device配置

<Device> system-view

[Device] nat address-group 0

[Device-address-group-0] address 4.4.4.25 4.4.4.30

[Device-address-group-0] quit

[Device] interface Route-Aggregation1023

[Device-Route-Aggregation1023] ip binding vpn-instance vpn11

[Device-Route-Aggregation1023] ip address 192.168.1.254 24

[Device-Route-Aggregation1023] quit

[Device] interface Route-Aggregation1021

[Device-Route-Aggregation1021] ip address 4.4.4.254 255.255.255.0

[Device-Route-Aggregation1021] nat outbound address-group 0

4. 故障现象

NAT业务不通，通过Probe视图下命令行display system internal ip packet-drop statistics查，发现有大量Match blackhole FIB 黑洞路由丢包统计，且不断快速增加。

[Device] display system internal ip packet-drop statistics slot 3 cpu                                                    

1                                                                                                                                  

CPU 1 on slot 3:                                                                                                                    

IPv4 packets dropping statistics:                                                                                                  

  Drop orignal paket after fragmentation:                       0                                                                   

  Match blackhole FIB:                                          230245                                                                  

  Interface forbids forwarding broadcast packets:               0                                                                   

  Fragments reassembly failed:                                  0                                                                  

  Fragment reassembly queue error:                              0                                                                   

  Fragments in queue reach the limit:                           0                                                                  

  Fragment overlapping:                                         0                                                                   

  fragmentation failed:                                         0                                                                  

  Invalid source IP address:                                    0                                                                   

  Receiving interface control block error:                      0                                                                  

  Sending interface control block error:                        0                                                                  

  Interface network status down:                                5588                                                               

  Unknown FIB forwarding type:                                  0                                                                  

  Drop layer 2 broadcast and multicast packets:                 0                                                                  

  TTL exceed:                                                   0                                                                  

  Unknown forwarding path:                                      0                                                                  

  No route:                                                     0                                                                  

  Insufficient memory:                                          0                                                                  

  Packet length less than 20 bytes:                             0                                                                  

  Unknown protocol type:                                        0                                                                   

  IP version error:                                             0                                                                  

  IP header length error:                                       0                                                                   

  Packet length less than that claimed in IP header:            0                                                                  

  Invalid destination IP address:                               0                                                                   

  IP options processing error:                                  0                                                                  

  IP checksum error:                                            0                                                                   

  Fragments in queue for virtual reassembly reach the limit:    0                                                                  

  Virtual fragment reassembly failed:                           0                                                                   

  Dropped by control plane policing:                            0                                                                  

  Expand packet buffer failed:                                  0                                                                   

  Packet buffer error:                                          0                                                                  

  Invalid fragment flag:                                        0                                                                  

  Packet length claimed in IP header larger than 65535 bytes:   0                                                                  

  Source or destination ip is loopback but not local:           0                                                                  

  Service processing error:                                     0                                                                  

  Search session failed:                                        0                                                            

1.1.2  故障处理步骤

1. 首先确认nat outbound的配置是否正确

[Device] display nat outbound                                                                                                      

NAT outbound information:                                                                                                          

  Totally 1 NAT outbound rules.                                                                                                     

  Interface: Route-Aggregation1021                                                                                                

  usCfgSeq: 1                                                                                                                       

    ACL: ---                                                                                                                      

    Address group ID: 257                                                                                                           

    Port-preserved: N        NO-PAT: N  Reversible: N                                                                              

    NAT counting: 0                                                                                                                

    Config status: Active                                                                                                           

Global flow-table status: Active  

检查Global flow-table status是否为Active，若为Inactive，建议删除此条nat配置后重新配置一次。

2. 打开debugging nat packet acl，确认debugging信息是否正确，应有类似如下debugging信息：

使用debugging命令时，请配置合理的ACL，避免输出过多的干扰日志不利于排查错误。

 *Dec 13 09:58:48:082 2013 H3C NAT/7/COMMON: -Chassis=2-Slot=10.1;

 PACKET: (Route-Aggregation1021-out) Protocol: TCP

       192.168.1.2:13249 -         4.4.4.6:   21(VPN:   16) ------>

        4.4.5.11:11000 -         4.4.4.6:   21(VPN:    0)

*Dec 13 09:58:48:083 2013 H3C NAT/7/COMMON: -Chassis=2-Slot=10.1;

 PACKET: (Route-Aggregation1021-in) Protocol: TCP

         4.4.4.6:   21 -        4.4.5.11:11000(VPN:    0) ------>

            4.4.4.6:   21 -     192.168.1.2:13249(VPN:   16)

注： ### 可以看到正向的流量做了NAT转换，从vpn11的域转成了没有vpn的域。

3. 通过display session table ipv4 verbose命令，确认会话是在哪一块引擎上建立的。

<Device> display session table ipv4 verbose

Slot 0 in chassis 1:

Total sessions found: 0

 

Slot 3 in chassis 1:

Total sessions found: 0

 

CPU 0 on slot 4 in chassis 1:

Total sessions found: 0

 

Slot 6 in chassis 1:

Initiator:

  Source      IP/port: 192.168.1.2/13790

  Destination IP/port: 4.4.4.6/21

  DS-Lite tunnel peer: -

  VPN instance/VLAN ID/VLL ID: vpn11/-/-

  Protocol: TCP(6)

Responder:

  Source      IP/port: 4.4.4.6/21

  Destination IP/port: 4.4.4.27/1060

  DS-Lite tunnel peer: -

  VPN instance/VLAN ID/VLL ID: vpn12/-/-

  Protocol: TCP(6)

State: TCP_ESTABLISHED

Application: FTP

Start time: 2013-12-15 10:49:00  TTL: 3592s

Interface(in) : Route-Aggregation1023

Interface(out): Route-Aggregation1021

Zone(in) : Trust

Zone(out): menglei

Initiator->Responder:            3 packets        128 bytes

Responder->Initiator:            2 packets        130 bytes

4. 查看openflow表项，确认表项是否和会话表项一致。

对于动态NAT，NAT表项会下刷到每一块业务板上，起到分流作用。

<Device> system-view

[Device] probe

[Device-probe] display system internal openflow instance inner flow-table

 Flow entry rule 6 information:

 cookie: 0x0, priority: 7301, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG1021

 Ethernet type: 0x0800

 IP Range: IPv4 destination address from 4.4.4.25 to 4.4.4.27

Instruction information:

 Write actions:

  Output interface: Blade2/4/0/1

Flow entry rule 7 information:

 cookie: 0x0, priority: 7301, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG1021

 Ethernet type: 0x0800

 IP Range: IPv4 destination address from 4.4.4.28 to 4.4.4.30

Instruction information:

 Write actions:

Output interface: Blade2/10/0/1

5. 经过以上排查步骤，可以再次检查nat业务是否正常。

在Probe视图下使用命令display system internal ip packet-drop statistics发现Match blackhole FIB黑洞路由丢包统计不再增加，说明业务正常。

6. 如果上述定位手段均不能作出结论，请联系相关技术支持人员协助分析

1.2  静态NAT444转换故障

1.2.1  故障描述

图1-2 静态NAT444转换组网图

2. 组网需求

Host 1访问Host 2，在Device上对Host 1的地址进行静态NAT444转换，转换公网地址池为：4.4.5.11到4.4.5.13。Device上有2块防火墙业务板。

3. Device配置

# 配置NAT444地址池。

<Device> system-view

[Device] nat port-block-group 256

[Device-port-block-group-256] local-ip-address 192.168.1.2 192.168.1.11 vpn-instance vpn11

[Device-port-block-group-256] global-ip-pool 4.4.5.11 4.4.5.12

[Device-port-block-group-256] block-size 1000

[Device-port-block-group-256] port-range 10000 19000

# 配置入接口。

<Device> system-view

[Device] interface Route-Aggregation1023

[Device-Route-Aggregation1023] ip binding vpn-instance vpn11

[Device-Route-Aggregation1023] ip address 192.168.1.254 24

# 配置出接口。

<Device> system-view

[Device] interface Route-Aggregation1021

[Device-Route-Aggregation1023] ip address 4.4.4.254 255.255.255.0

[Device-Route-Aggregation1023] nat outbound port-block-group 256

# 配置vpn-instance到公网之间路由。

略。

4. 故障现象

NAT444不能正常转换、NAT444转换的报文不能正常转发、反向报文无法正常转发。

1.2.2  故障处理步骤

1. 确认NAT444的地址和端口块设置的正确性

<Device> display nat port-block-group 256

  Port block group 256:

    Port range: 10000-19000

    Block size: 1000

    Local IP address information:

      Start address        End address          VPN instance

      192.168.1.2          192.168.1.11         vpn11

    Global IP pool information:

      Start address        End address

      4.4.5.11             4.4.5.12

2. 确认端口块数和公网地址是否满足私网地址的需求

这里，每一个私网需要的端口块的端口个数为：1000。

私网地址段192.168.1.2-192.168.1.11共有10个私网地址：共需要1个地址块。

端口范围设置为：10000-19999，因此每一个公网地址可以提供9个地址块。

因此，从上面的配置分析，10个私网地址需要2个公网地址，这里的设置满足需求。

3. 通过debugging nat packet，查看nat444转换debugging信息

4. 通过display session table ipv4 verbose命令，查看会话是否正确。

5. 查看openflow表项下发是否正确

<Device> system-view

[Device] probe

[Device-probe] display system internal openflow instance inner flow-table

Flow entry rule 24 information:

 cookie: 0x0, priority: 7521, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG1021

 Ethernet type: 0x0800

 IP Range: IPv4 destination address from 4.4.5.11 to 4.4.5.12

Instruction information:

 Write actions:

  Output interface: Blade2/10/0/1

 

Flow entry rule 25 information:

 cookie: 0x0, priority: 7500, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Ethernet type: 0x0800

 IP Range: IPv4 source address   from 192.168.1.2 to 192.168.1.11

 VRF index: 16

<Device> display ip vpn-instance instance-name

Instruction information:

 Write actions:

 Output interface: Blade2/10/0/1

 

Flow entry rule 26 information:

 cookie: 0x0, priority: 7501, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Ethernet type: 0x0800

 IP Range: IPv4 destination address from 192.168.1.2 to 192.168.1.11

 VRF index: 16

Instruction information:

 Write actions:

 Output interface: Blade2/10/0/1

分析：我们可以看到下发了三条openflow，对于静态nat444来说，所有的openflow都是下发到主板卡上去的。

可以通过 display blade-controller-team default 这条命令可以用来查看哪块是主板卡，如下：

<Device> display blade-controller-team Default

ID: 1    Name: Default

  Chassis    Slot    CPU    Status    LBGroupID

  2          3       1      Normal        1

* 2          4       1      Normal        1

                                       

* : Primary blade controller of the team.

 

下面分析一下三条openflow：

(1)     IP Range:IPv4 destination address from 4.4.5.11 to 4.4.5.11

这一条指明了从Host 2回到Host 1的流量（经过nat转换之后的地址）该上送到哪块板卡上。

(2)     IP Range:IPv4 source address from 192.168.1.2 to 192.168.1.2

这一条指明了从Host 1到Host 2的流量该上送到哪块板卡上。

(3)     IP Range:IPv4 destination address from 192.168.1.2 to 192.168.1.2

大家对这条openflow可能会不理解，觉得这条openflow为什么要下发呢？其实此时如果要是有一个Host 3（和Host 1同网侧）想要访问Host 1，那么Host 3访问Host 1的流量该上送到哪块板卡呢？由于第二条openflow的存在，Host 1的流量肯定上送到主板卡，如果Host 3访问Host 1的流量没有上送到主板卡，而上送到别的板卡上去了，那么Host 1此时就没法访问Host 3了。

6. 通过会话和openflow下刷表项的对比，如果存在不一致的地方，NAT444转换可能存在异常。如果这些都解决不了问题。请联系技术支持人员进行分析。

1.3  设备作为出口网关设备割接之后，NAT业务不通，但是接口地址可以ping通

1.3.1  故障描述

Device作为出口网关设备割接之后，内网部分用户无法上网，外网用户无法访问内网服务器，但是从外网ping出接口的地址可以ping通。

1.3.2  故障处理步骤

1. 确定NAT地址池是否和接口地址是同一个网段：

如果NAT地址池的地址和接口地址不在同一网段，NAT地址池的地址无法响应。如果不在同一网段，要确保对端设置了NAT地址池的路由。

2. 割接后，如果地址池中的地址或nat server地址和接口在同一网段，确认地址池中的地址或者nat server地址是否发送了免费ARP，可以通过直连对端设备进行确认。还需要确认对端学习到的arp的mac地址的正确性：

设备割接时，对端设备需要更新ARP。当两端不是直连，对端设备不能感知到链路Down过，所以不能删除相关ARP表项。当设备上线后，本端接口会发送接口地址的免费ARP，对端设备收到该免费ARP后可以正常更新该ARP表项；但可能存在地址池中的地址ARP没有刷新。

3. debug或者抓包分析，是否ping报文只有发出去的而没有回来的，存在转发异常的情况。

4. 持续地ping nat地址池或者nat server的地址，打开arp的debug开关，确认是否能够收到arp请求报文。

5. 如果无法确认定位，请联系技术支持人员进行分析。

1.4  IPv6访问IPv4（以源地址动态转换，目的地址静态转换为例）

1.4.1  故障描述

图1-3 IPv6访问IPv4组网图

 

2. 组网需求

Host 1访问Host 2。在Device上，通过IPv4到IPv6源地址静态转换策略，为目的IPv4地址指定一个对应的IPv6地址23::1。Host 1访问该IPv6地址便可以访问Host 2。

对于Host 1，通过IPv6到IPv4的源地址动态转换策略，将IPv6发送过来的IPV6报文源地址转换为IPv4地址30.30.40.100。

3. Device配置

<Device> system-view

[Device] acl ipv6 number 2000

[Device-acl-ipv6-basic-2000] rule 0 permit source 1:1::1/128

[Device-acl-ipv6-basic-2000] quit

[Device] aft address-group 0

[Device-aft-address-group-0] address 30.30.40.100 30.30.40.100

[Device-aft-address-group-0] quit

[Device] aft v6tov4 source acl ipv6 number 2000 address-group 0

[Device] aft v4tov6 source 1.1.1.1 23::1

[Device] interface Route-Aggregation900

[Device-Route-Aggregation900] aft enable

[Device-Route-Aggregation900] quit

[Device] interface Route-Aggregation901

[Device-Route-Aggregation901] aft enable

4. 故障现象

AFT不能正常转换或者AFT转换的报文不能正常转发。

1.4.2  故障处理步骤

1. 首先确认AFT配置是否正确

Display aft configuration查看设备上AFT的配置。在Device上，流量入接口和出接口都需要开启aft功能（aft enable）。

<Device> display aft configuration

aft address-group 0

 address 30.30.40.100 30.30.40.100

 

aft v6tov4 source acl ipv6 number 2000 address-group 0

 

aft v4tov6 source 1.1.1.1 23::1

 

interface Route-Aggregation10.900

 aft enable

interface Route-Aggregation10.901

 aft enable

 

AFT ALG:

  DNS        : Enabled

  FTP        : Enabled

  HTTP       : Enabled

  ICMP-ERROR : Enabled

  RTSP       : Enabled

  SIP        : Enabled

2. Debugging AFT 事件查看AFT是否正常转换

<Device> debugging aft packet ip

Dec 16 15:08:22:697 2020 H3C AFT/7/COMMON: -Slot=6.1;

 PACKET: (Route-Aggregation10.900) Protocol: UDP

 1.1.1.1/69 - 30.30.40.100/1128(VPN:0) ------>

 23::1/69 – 1:1::1/35017(VPN:0)

或

<Device> debugging aft packet ipv6

Dec 16 15:09:13:696 2020 H3C AFT/7/COMMON: -Slot=6.1;

 PACKET: (Route-Aggregation10.901) Protocol: UDP

 1:1::1/6677 - 23::1/5060(VPN:0) ------>

 30.30.40.100/1149 - 1.1.1.1/5060(VPN:0)

注：如果出现以上信息，表示IPv4和IPv6已经进行了AFT转换。

3. 查看openflow表项，确认流表下发是否正常

<Device> system-view

[Device] probe

[Device-probe] display system internal openflow instance inner-redirect flow-table

Flow entry 3305 information:

 cookie: 0x0, priority: 5045, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG10

 VLAN ID: 900, mask: 0xfff

 IP Range: IPv4 destination address from 30.30.40.100 to 30.30.40.100

Instruction information:

 Write actions:

  Group: 4026531857

 

Flow entry 3306 information:

 cookie: 0x0, priority: 5045, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG10

 VLAN ID: 4094, mask: 0xfff

 IP Range: IPv4 destination address from 30.30.40.100 to 30.30.40.100

Instruction information:

 Write actions:

  Group: 4026531857

 

Flow entry 3307 information:

 cookie: 0x0, priority: 5080, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 IPv4 source address: 1.1.1.1, mask: 255.255.255.255

Instruction information:

 Write actions:

  Group: 4026531865

 

Flow entry 3308 information:

 cookie: 0x0, priority: 5085, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 IPv4 destination address: 1.1.1.1, mask: 255.255.255.255

Instruction information:

 Write actions:

  Group: 4026531865

 

Flow entry 3309 information:

 cookie: 0x0, priority: 7085, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG10

 VLAN ID: 900, mask: 0xfff

 IPv6 destination address: 23::1

 IPv6 destination address mask: FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Instruction information:

 Write actions:

  Group: 4026531865

 

Flow entry 3310 information:

 cookie: 0x0, priority: 7085, hard time: 0, idle time: 0, flags: check_overlap

 |reset_counts|no_pkt_counts|no_byte_counts, byte count: --, packet count: --

Match information:

 Input interface: RAGG10

 VLAN ID: 4094, mask: 0xfff

 IPv6 destination address: 23::1

 IPv6 destination address mask: FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Instruction information:

 Write actions:

  Group: 4026531865

注：对于AFT静态转换，需要关注流表下发是否正常。

4. 如果上述定位手段均不能作出结论，请联系相关技术支持人员协助分析

1.5  故障诊断命令

表1-1 故障诊断命令

命令	说明
display nat outbound	显示nat outbound设置信息
display nat server	显示nat server设置信息及状态
display blade-controller-team Default	显示设备上那块业务板为主业务板
display openflow instance	显示openflow下刷的表项
display session	显示会话信息
save	将当前配置保存到指定文件