A hash algorithm converts an input of any length into an output of a fixed length. This output is called a hash value. The space of a hash value is usually much smaller than the input space. Different inputs might be converted to the same output. An output cannot be reversed to identify its input.

The hash algorithm typically applies to the following applications on switches:

· Hash bucket calculation for storing table entries.

· Route selection for load sharing.

This document introduces the route selection for load sharing of packets and data flows. Load sharing route selection includes IP packet Equal-Cost Multi-Path (ECMP) route selection and aggregation member port route selection.

About load sharing

ECMP routes are multiple routes discovered by the same routing protocol with the same destination IP address and cost value. Traffic matching ECMP routes can be load balanced across multiple paths. This can increase transmission bandwidth and implement data backup from failed links without delay or packet loss, enhancing network reliability.

Ethernet link aggregation can increase link bandwidth by aggregating multiple physical Ethernet links with a single logical link. The link reliability is effectively enhanced through dynamic backup among the member links. When forwarding packets, the aggregate interface can implement load sharing among multiple aggregation member ports.

Figure 1 Load sharing for ECMP routes and aggregate links

Load sharing can be implemented on a per-packet basis or per-flow basis.

· Per-packet load sharing—The device forwards packets to different forwarding paths in the order the packets are received. In this mode, packets in the same flow might be forwarded through different paths, resulting in packet disorder on the receiver. To avoid this issue, make sure the receiver supports reordering disordered packets.

· Per-flow load sharing—The device identifies a flow based on the source IP address, destination IP address, source port number, destination port number, IP protocol number, and incoming port. In per-flow load sharing, the device forwards packets of one flow through the same link.

Figure 2 Par-packet load sharing

Figure 3 Per-flow load sharing

Hash algorithms for load sharing

A switch implements load sharing through a hash algorithm as follows:

1. Obtain the input value, such as the hash factor.

2. Perform hash calculation.

3. Identify the ECMP route or outbound aggregate interface based on the hash calculation result.

Data center switch series can be divided into the following groups. The switch series in the same group use similar hash algorithms.

· S12500X-AF, S12500F-AF, S12500R (type H cards and type K cards), and S6890.

· S6800, S6860, S6805, S6825, S6850, S9850, and S9820.

Load sharing algorithms for S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards and type K cards), and S6890 switch series

Hash calculations for the S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards and type K cards), and S6890 switch series are as follows:

1. Select packet information (such as the source MAC address, destination MAC address, source IP address, destination IP address, protocol number) as hash factors based on the device configuration.

2. Perform calculation based on hash factors and the hash algorithm, and generate a 16-bit hash key.

3. Set the shift value for the hash key. Hash keys calculated based on specific traffic characteristics vary little on higher digits, which causes hash polarization. To solve this issue, set the shift value to circular shift the hash key by certain bits.

4. Calculate the number of outbound interfaces by using offset = (key * size) >> 16. The size value indicates the number of outbound interfaces, which can be ECMP routes or aggregation ports. The >> 16 part means to right circular shift the hash key by 16 bits.

Figure 4 Load sharing algorithms for the S12500X-AF, S12500F-AF, S12500R (type H cards and type K cards), and S6890 switch series

Table 1 Description of load sharing algorithms

Algorithm number for command lines

Description

ECMP 0 to 5

Aggregation 1 to 6

The six hash algorithms of the chip cannot accurately correspond to traffic characteristics. When hash imbalance occurs, you can try another algorithm.

ECMP 0 and Aggregation 1 indicate the same algorithm. ECMP 1 and Aggregation 2 indicate the same algorithm, and so on. To avoid hash polarization, do not configure the same algorithm for ECMP and link aggregation.

ECMP 6 to 9

Aggregation 7 to 10

These algorithms overlap with previous ones. The chip supports only six algorithms. The value range for arguments in the commands remains unchanged for compatibility with older products and earlier versions.

The default shift values for these algorithms differ from those of the previous algorithms, and the hash calculation results are different either. When hash imbalance occurs, you can try one of these algorithms.

ECMP 10

Aggregation 11

The per-packet hash algorithm might cause packet disorder. As a best practice, do not specify this algorithm.

ECMP 11 to 15

These algorithms are reserved for testing scenarios where only one hash factor is set and the number of ECMP routes is 2, 4, 8, 16, or 32.

As a best practice, do not specify these algorithms.

NOTE:

ECMP x indicates the algorithm specified by using the ip load-sharing mode per-flow algorithm x command. Aggregation x indicates the algorithm specified by using the link-aggregation global load-sharing algorithm x command.

Load sharing algorithms for S6800, S6860, S6805, S6825, S6850, S9850, and S9820 switch series

Hash calculations for the S6800, S6860, S6805, S6825, S6850, S9850, and S9820 switch series are as follows:

1. Select the field for hash calculation (Field Selection) based on the EtherType field in the packet. Field Selections are divided into two types. Field Selection A is used for link aggregation, and Field Selection B is used for ECMP.

Both Field Selection A and Field Selection B can generate a set composed of 13 hash bins, which contains the packet characteristics used in the hash calculation. Each hash bin is 16 bits long. These 13 hash bins take 208 bits.

2. Calculate ECMP hash results (Hash B0 and Hash B1) and link aggregation hash results (Hash A0 and Hash A1) based on the 208-bit hash bins and the specified load sharing algorithm.

The hash algorithm cannot accurately correspond to traffic characteristics. When hash imbalance occurs, you can adjust the hashing algorithm, seed value, or shift value to update Hash A0, Hash A1, Hash B0, and Hash B1.

3. As shown in Figure 6, the Hash Key (83 bits) consists of Hash A0 (16 bits), Hash A1 (16 bits), Hash B0 (16 bits), Hash B1 (16 bits), LBN (4 bits, obtained through the ingress port), Destination Port (7 bits, H3C reserved), and Load Balance ID (LBID, 8 bits). The device selects a subset from these 83 bits to calculate the outbound interfaces.

4. Calculate the number of outbound interfaces by using offset = ((hash value & 65535) % (ecmp_count + 1)) & 0x3FF. The hash value & 65535 portion is used to make sure the hash value is within the value range. The & 0x3FF portion is used to make sure the outbound interface number is within the value range.

Figure 5 Load sharing algorithms for the S6800, S6860, S6805, S6825, S6850, S9850, and S9820 switch series

Figure 6 Hash key calculation for the S6800, S6860, S6805, S6825, S6850, S9850, and S9820 switch series

Commands for adjusting hash-based load sharing

When load imbalance occurs, you can adjust hash calculation results by changing the hash algorithm, seed value, and shift value.

Table 2 Commands for adjusting hash-based load sharing

Item	ECMP hashing	Aggregation hashing
Hash factor	ip load-sharing mode	For the S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards and type K cards), and S6890 switch series: · link-aggregation global load-sharing mode (system view) · link-aggregation load-sharing mode (interface view) For the S6800, S6860, S6805, S6825, S6850, S9850, S9820-64H, S9820-8C switch series: · Method 1: link-aggregation load-sharing ignore · Method 2 (not for broadcast packets): ¡ link-aggregation global load-sharing mode (system view) ¡ link-aggregation load-sharing mode (interface view) After adjusting the hash factors by using method 2, the hash calculation process for aggregation load balancing will change significantly. The new calculation process is not recommended. As a best practice, use method 1 to adjust hash factors.
Hash algorithm	ip load-sharing mode per-flow algorithm	link-aggregation global load-sharing algorithm
Seed value (When the network has devices from multiple vendors, as a best practice, configure the same seed value for all devices.)	ip load-sharing mode per-flow algorithm algorithm-number seed seed-number	link-aggregation global load-sharing seed
Shift value (When the network has devices from multiple vendors, as a best practice, configure the same shift value for all devices.)	ip load-sharing mode per-flow algorithm algorithm-number shift shift-number	link-aggregation global load-sharing offset The offset value specified in this command is the shift value in Figure 4. This command is not supported on the S12500R switch series.

Table 3 Default values and value ranges for hash-based load sharing algorithms on the S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards and type H cards), and S6890 switch series

Item

Default value

Value range

Hash factor

For ECMP:

dest-ip, dest-port, ingress-port (not supported on the S12500R switch series), ip-pro, src-ip, src-port, and flow-lab

For link aggregation:

The device automatically determines the load sharing mode according to the packet type.

When forwarding Layer 2 packets:

· If the packets have IP headers, the device uses load sharing modes of the source MAC address, destination MAC address, source IP address, destination IP address, source port, destination port, and protocol number.

· If the packets have no IP headers, the device uses load sharing modes of the source MAC address, and destination MAC address.

When forwarding Layer 3 packets, the device uses load sharing modes of the source IP address, destination IP address, source port, destination port, and protocol number.

For ECMP:

dest-ip, dest-port, flow-label, ingress-port (not supported on the S12500R switch series), ip-pro, src-ip, and src-port

For aggregation:

destination-ip, destination-mac, destination-port, ingress-port (not supported on the S12500R switch series), ip-protocol, source-ip, source-mac, and source-port

Hash algorithm

For ECMP: 0

For link aggregation: 4

For ECMP: 0 to 15

For link aggregation: 1 to 11

Seed value

For ECMP: 0

For link aggregation: 0x1

For ECMP: 0-FFFF

For link aggregation: 0-FFFF

Shift value

For ECMP: 0

For link aggregation: 0

For ECMP: 0 to 15

For link aggregation: 0 to 15

For the S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards, type K cards) switch series, both ECMP load sharing and link aggregation load sharing use the same five-tuple register. Therefore, the most recent configuration takes effect. As a best practice, configure the same five-tuple information for both ECMP and link aggregation load sharing.

Table 4 Default values and value ranges for hash-based load sharing on S6800, S6860, S6805, S6825, S6850, S9850, S9820-64H, and S9820-8C switch series

Item

Default value

Value range

Hash factor

For ECMP:

dest-ip, dest-port, ingress-port, ip-pro, src-ip, and src-port

For link aggregation:

· For Layer 2 packets: destination-mac address, source-mac address, and ethernet-type

· For Layer 3 packets: destination-ip address, source-ip address, destination-port, source-port, and ip-protocol

For ECMP:

dest-ip, dest-port, flow-label, ingress-port, ip-pro, src-ip, and src-port

For link aggregation:

destination-ip, destination-mac, destination-mac, destination-port, ingress-port, mpls-label1, mpls-label2, source-ip, source-mac, and source-port

For the S6805, S6825, S6850, S9850, and S9820-64H switch series, the configuration of hash factors take effect only for known unicast packets.

Hash algorithm

For ECMP:

· For the S6800 and S6860 switch series: Before any configuration, the algorithm number 8 is specified. If you execute undo ip load-sharing mode after specifying an algorithm, the algorithm number 0 is specified. To view the currently effective algorithm, use the display ip load-sharing mode command.

· For the S6805, S6825, S6850, S9850, and S9820 switch series: 8

For link aggregation:

· For the S6800 and S6860 switch series: 5

· For the S6805, S6825, S6850, S9850, and S9820 switch series: 0

For ECMP:

· For the S6800 and S6860 switch series: 0 to 8

· For the S6805, S6825, S6850, S9850, and S9820 switch series: 0 to 13

For link aggregation:

· For the S6800 and S6860 switch series: 1 to 8

· For the S6805, S6825, S6850, S9850, and S9820 switch series: 1 to 13

Seed value

For ECMP: 0

For link aggregation: 0x0

For ECMP: 0 to FFFFFFFF

For link aggregation: 1 to FFFFFFFF

Shift value

For ECMP: 0

For link aggregation: 0

For ECMP:

· For the S6800 and S6860 switch series: 0 to 63

· For the S6805, S6825, S6850, S9850, and S9820 switch series: 0 to 15

For link aggregation: 1 to 63

Configuring ECMP and link aggregation load sharing modes based on traffic type

As a best practice, configure the load sharing mode for ECMP and link aggregation based on different packet types, as shown in Table 5.

Table 5 Load sharing modes for different packet types

Packet type	Recommended load sharing mode
IPv4/IPv6 packets	Typically, use the default hash factor configuration. If hash imbalance occurs, analyze traffic characteristics and specify items with rich variations as hash factors.
Multicast IPv4/IPv6 packets	ECMP hashing configuration does not affect multicast packets. For link aggregation hashing, you can change the hash algorithm. In most cases, use the default configuration.
MPLS packets	For ingress and egress nodes: Use original packet information. For transit nodes: For the S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards), and S6890 switch series: Use the MPLS label, IP information, and L4 port information. Layer 2 packets do not support hashing. For the S6800, S6860, S6805, S6825, S6850, S9850, S9820-64H, and S9820-8C switch series: Use the MPLS label and IP information. For Layer 2 packets, use the MPLS label. NOTE: For S12500R (type K cards) switches operating as transit nodes, the load sharing mode remains to be verified.
Layer 2 packets other than IPv4, IPv6, and MPLS packets	destination-mac address, source-mac address, and ethernet-type or destination-mac address and source-mac address
Tunnel packets (such as GRE packets)	Encapsulated packets, such as GRE and IPinIP packets, have fixed outer headers. For example, the source and destination IP addresses are fixed at the IP addresses of the loopback interfaces on the tunnel source end and destination end, respectively. Load sharing cannot appropriately be performed by outer IP packet information. Use the following commands to perform load sharing for tunnel packets by inner IP header information or both the inner and outer IP header information: For ECMP: ip load-sharing mode per-flow tunnel { all \| inner \| outer } For link aggregation: link-aggregation global load-sharing tunnel { all \| inner \| outer } Support for the all keyword varies by device model.
VXLAN packets	Use the original packet information for encapsulation and decapsulation devices. Use the outer or inner layer information for VXLAN packets being forwarded. To adjust the hash algorithm for VXLAN packets being forwarded, use the following commands: For ECMP: ip load-sharing mode per-flow tunnel { all \| inner \| outer } For link aggregation: link-aggregation global load-sharing tunnel { all \| inner \| outer } Support for the all keyword varies by device model. For the S12500X-AF (type H cards), S12500F-AF, S12500R (type H cards, type K cards), and S6890 switch series: Specify the all keyword for hashing of VXLAN packets being forwarded.

Troubleshooting hash-based load imbalance

About load imbalance

Load sharing aims to evenly distribute traffic to multiple ECMP routes or aggregation member ports. Load imbalance refers to scenarios where traffic is distributed to one or multiple forwarding paths and other paths have little or even no traffic. Load imbalance reduces link utilization and might cause errors on links with heavy traffic load.

General principles for resolving load imbalance

Adjust load sharing for ECMP or link aggregation on the local device. If local adjustment cannot resolve the issue, adjust the ECMP hash algorithm on upper-level devices, and pay attention to the impact on lower-level devices.

As a best practice, perform the following tasks in order:

1. Resolve hash polarization. For more information about hash polarization, see "Hash polarization."

2. Adjust hash factors.

Specify items with rich variations as hash factors. For example, if the MAC addresses of the packets change significantly but the load-sharing method is set to source-ip, load sharing cannot be implemented. Additionally, identify whether the traffic enters from multiple ports or a single port. If it enters from multiple ports, you can add ingress port as a hash factor.

3. Adjust the hash algorithm.

4. Adjust the seed value.

5. Adjust the shift value.

6. Adjust the number of member ports. As a best practice, make sure the number of member ports is a power of two.

For more information about configuration of the hash factor, hash algorithm, seed value, and shift value, see Table 2.

Hash polarization

About hash polarization

Hash polarization refers to hash imbalance that occurs after hash calculation is performed on the traffic for two or more times. Hash polarization might occur in the following situations:

· The level-1 devices perform ECMP hashing, and the level-2 devices also perform ECMP hashing.

· The level-1 devices perform ECMP hashing, and the level-2 devices perform link aggregation hashing.

· Outbound interfaces for one or multiple ECMP routes on devices at the same level are aggregate interfaces. The traffic undergoes ECMP hashing and then link aggregation hashing.

As shown in Figure 7, SW1 receives two flows and evenly distributes the flows to two outbound interfaces through hashing. If SW2 uses the same or similar load sharing mode as that of SW1, SW2 might unevenly distribute the traffic. Most or all of traffic 1 (in green) goes through several outbound interfaces and other interfaces have little or no traffic. The situation on SW3 is similar.

Figure 7 Hash polarization

Avoiding hash polarization

1. In multi-level networks, do not use devices with the same or similar hash algorithms at adjacent levels.

Data center switch series can be divided into the following groups. The switch series in the same group use similar hash algorithms.

¡ S12500X-AF, S12500F-AF, S12500R (type H cards and type K cards), and S6890.

¡ S6800, S6860, S6805, S6825, S6850, S9850, and S9820.

For example, if you use S6850 switches at level 2 and S12500X-AF switches at level 1, the hash polarization risk is low. If you use S6850 switches at level 1 and S9820-8C switches at level 2, the hash polarization risk is high.

2. In multi-level networks, specify different hash algorithms and hash factors for level-1 and level-2 devices. For example, configure the level-1 devices to use algorithm 1, and the level-2 devices to use algorithm 4. Configure the level-1 devices to perform load sharing based on the source IP address, and the level-2 devices to perform load sharing based on the destination IP address.

3. Make sure the numbers of outbound interfaces for the first hashing and the second hashing are coprime.

For example: if the level-1 devices perform link aggregation hashing with two physical outbound interfaces, make sure the level-2 devices perform ECMP hashing with three outbound interfaces.

Enabling status of local-first load sharing across IRF member devices

The forwarding process for ECMP or link aggregation load sharing across IRF member devices is as shown in Figure 8. When local-first load sharing is enabled for ECMP or link aggregation load sharing, the traffic load is balanced among local outbound interfaces but not balanced among outbound interfaces of different member devices. When local-first load sharing is disabled for ECMP or link aggregation load sharing, the traffic load is balanced among outbound interfaces of all member devices. However, the load of IRF physical links will increase because some packets are forwarded through IRF links.

On DC switches, local-first load sharing is enabled by default for ECMP and link aggregation.

You can adjust the enabling status of local-first load sharing for ECMP and link aggregation by using the following commands:

· ip load-sharing local-first enable (system view)

· link-aggregation load-sharing mode local-first (system view)

Figure 8 Forwarding process for ECMP or link aggregation load sharing across IRF member devices

Increasing the number of member interfaces to be a power of two

Increasing the number of ECMP routes or aggregation member ports will change the load sharing results. As a best practice to increase the number of interfaces, make sure the total number of interfaces is a power of two. This can better match the packet sending interval and thus make load sharing more evenly.

Other common hash functions

Enhanced ECMP mode

The enhanced ECMP mode and normal ECMP mode use the same hash algorithm to select routes per flow, but differ in processing an ECMP route failure.

· The normal ECMP mode enables the device to reallocate all traffic to the remaining routes. This might cause changes to the forwarding paths of traffic and thus interrupting services that require session persistence.

· The enhanced ECMP mode enables the device to reallocate only the traffic of the failed route to the remaining routes. This protects traffic on normal links from being affected.

Figure 9 Load sharing when links are normal

Figure 10 Traffic reallocation in normal ECMP mode upon link failure

Figure 11 Traffic reallocation in enhanced ECMP mode upon link failure

To enable the enhanced ECMP mode, use the ecmp mode enhanced in system view.

Resilient mode for link aggregation load sharing

The resilient mode enables the device to redistribute as less traffic as possible when a link state change occurs to minimize its impact on services.

For example, an aggregation group has three member links with load sharing enabled. When a link fails, the following situations can occur:

· If the resilient mode is disabled, the system rehashes the traffic across the two remaining links.

· If the resilient mode is enabled, the system rehashes only the traffic on the failed link across the two remaining links. Because the existing traffic on the two remaining links is not reallocated, impact on the ongoing services is minimized.

When the failed link recovers, the system rehashes part of the traffic on the other two links to the recovered link. Because not all traffic is rehashed, the traffic distribution pattern might differ from what it was before the link failure.

In this mode, an aggregation group distributes traffic based on the default load sharing mode when no link change occurs.

To set the resilient mode for link aggregation load sharing, use the link-aggregation load-sharing mode resilient command in aggregate interface view.

The feature is supported only on the S6800, S6860, S6805, S6825, S6850, and S9850 switch series.

Symmetric load sharing

Symmetric load sharing ensures that bidirectional traffic specific to a particular source and destination address pair flow along the same path.

For example, two firewalls are attached to a switch for traffic security. The switch hashes traffic to balance the load to the firewalls. Each firewall establishes a session to process half of the traffic. If the hashing result for the incoming traffic is inconsistent with that of the outgoing traffic, the firewall will re-establish sessions. In this case, you can enable symmetric load sharing to save firewall resources and ensure accurate protection for traffic.

Figure 12 Outgoing traffic hashing for two firewalls attached to one switch

Figure 13 Incoming traffic hashing for two firewalls attached to one switch

To enable symmetric load sharing, use the ip load-sharing symmetric enable in system view.

Dynamic load sharing

Static load sharing is implemented based on specified traffic characteristics and cannot change with actual forwarding situations and link loads. For example:

· Traffic with the same characteristics is always distributed to a fixed path, even if the data flow matching the characteristics is no longer transmitted.

· Load imbalance might occur when traffic sizes differ significantly, because the actual link loads are ignored.

Dynamic load sharing uses timers and real-time load measurement (port bandwidth load and queue size) to optimize load sharing results and implement intelligent hashing.

Dynamic load sharing modes include the following:

· Eligibility mode.

· Spray mode.

· Fixed mode.

In the eligibility mode, the device redistributes the packets in one flowset from heavy-load links to the link with the lightest load for load sharing. One flowset refers to the packets to be forwarded in a flow within one flowset inactive time interval. The eligibility mode enables the device to forward all packets in one flowset through the same path. The detailed process is as follows:

1. When traffic with specific characteristics enters a device for the first time, it is considered a new flow. The device establishes a flowset for the traffic and set a flowset inactive timer.

2. The device calculates the path with a lighter load within the flowset inactive time interval. It uses the same forwarding path for traffic with the same characteristics within the interval. It also refreshes the flowset inactive timer in real time to maintain the flowset in an active state (session persistence).

3. If no traffic maintains the active state of the flowset before the flowset inactive timer expires, the flowset becomes aged. The device will create a new flowset for subsequent traffic with the same characteristics, and rehashes it to a path with a lighter load.

Figure 14 Eligibility-mode load sharing

The spray mode enables the device to implement load sharing on a per-packet basis. In this mode, the device forwards a packet through the path with lightest load. Packets in the same flow might be forwarded through different paths, resulting in packet disorder on the receiver. To avoid this issue, make sure the receiver supports reordering disordered packets.

Figure 15 Spray-mode load sharing

The fixed mode enables the device to forward all packets in one flow through the same path. When the device forwards the first packet of a flow, it selects a path with a lighter load.

Figure 16 Fixed-mode load sharing

To configure dynamic load sharing for ECMP, use the ecmp mode { eligible [ flowset-inactive-time flowset-inactive-time ] | fixed | spray } command. This command is supported only on the S6805, S6825, S6850, S9850, S9820-64H, and S9820-8C switch series.

To configure dynamic load sharing for link aggregation, use the link-aggregation load-sharing mode dynamic { eligible [ flowlet-gap-time flowlet-gap-time ] | fixed | spray } command. This command is supported only on the S6805, S6825, S6850, S9850, S9820-64H, and S12500R switch series.

H3C Hash-Based Load Sharing on H3C DC Switches-6W100

Enabling status of local-first load sharing across IRF member devices

Intelligent Terminal Products

Product Support Services

Technical Service Solutions

Resource Center

Policy

Online Help

Become a Partner

Partner Policy & Program

Global Learning

Partner Sales Resources

Service Business

News & Events

Contact Us