FCoE Technology White Paper

    16-08-2018

H3C_彩色.emf



Overview

A data center using the FC SAN technology usually contains separate LANs and SANs. LANs transport traditional Ethernet/IP services, and SANs transport network storage services.

To provide both IP services through LANs and storage services through SANs, data center servers use independent Ethernet adapters and FC adapters to access LANs and SANs, respectively. Such an implementation entails a large number of adapters, cables, and devices, and brings high maintenance costs.

Fibre Channel over Ethernet (FCoE) consolidates LANs and SANs to reduce the number of adapters, cables, and switching devices in the data center. This reduces costs.

FCoE transports FC over Ethernet. In an FCoE solution, the server uses an FCoE-capable Ethernet adapter, and the FCoE forwarder (FCF) switches replace the traditional IP switches and FC switches. As a result, the number of network adapters, switches, and cables is greatly reduced, and the network operation and maintenance workload is reduced. In all, the total cost is reduced.

Figure 1 FCoE for I/O consolidation

As shown in Figure 1, in the traditional network:

The server is connected to the LAN through an Ethernet interface.

The server is connected to the SAN through an FC interface.

In the FCoE network:

The server is connected to the FCF switch.

The FCF switch is connected to the LAN through an Ethernet interface and to the SAN through an FC interface.

The link between the server and the FCF switch can transmit both Ethernet frames and FC frames.

FCoE implementation

Concepts

FCF

A switch that supports FC and FCoE is called an FCF.

ENode

An FCoE-capable node (server or storage) is called an ENode.

FCM

When an FC frame enters an Ethernet network, FCoE encapsulates it into an Ethernet frame for transmission. When the FC frame leaves the Ethernet network, FCoE decapsulates it from the Ethernet frame for further forwarding. This encapsulation/decapsulation process is called FC mapping (FCM).

FPMA

In FCM encapsulation, an ENode is assigned a fabric-provided MAC address (FPMA).

Figure 2 FPMA composition

As shown in Figure 2, in an FPMA:

The 24 most significant bits is the FC-MAP.

The 24 least significant bits is the ENode's FC ID.

The FC-MAP is designed to generate FPMAs and is configurable. It is in the range of 0x0EFC00 to 0x0EFCFF and is 0x0EFC00 by default. The FC ID is assigned to the ENode when the ENode performs a fabric login.

VFC

In an FC network, two FC switches are connected through a point-to-point link. In an Ethernet network, two FCFs are connected through a multi-access link. Therefore, a virtual point-to-point link is required between FCFs in an FCoE network. It ensures that the FC protocol can run on the same topology as in a native FC network. A virtual FC (VFC) link is a point-to-point link that connects two VFC interfaces on two FCoE entities.

FIP

The FCoE Initialization Protocol (FIP) sets up and maintains VFC links.

DCE

Data Center Ethernet (DCE), also called Converged Enhanced Ethernet (CEE), ensures lossless transmission of FC frames over Ethernet by introducing priority-based flow control (PFC) and bandwidth management.

DCBX

The Data Center Bridging Exchange (DCBX) protocol exchanges the capabilities of DCE devices and configure parameters for peers.

PFC

In Ethernet, FCoE traffic and other traffic are differentiated by priority. Priority-based Flow Control (PFC) enforces flow control for FCoE traffic identified by its priority to ensure no loss of packets. Other traffic is not imposed with flow control.

ETS

Scheduling bandwidth in a round robin way (WRR), Enhanced Transmission Selection (ETS) ensures that the bandwidth on an Ethernet link is shared by FCoE traffic and other traffic.

Mechanism

FIP

FIP sets up VFC links between FCFs and ENodes. Then, FC can run on established VFC links.

FIP packets include the following types:

FIP Discovery Solicitation.

FIP Discovery Advertisement.

Virtual Link Instantiation Request.

Virtual Link Instantiation Reply.

FIP Keep Alive.

FIP Clear Virtual Links.

FIP VLAN Request.

FIP VLAN Notification.

Figure 3 shows a typical process for FIP to set up a virtual link between an ENode and an FCF.

Figure 3 Setting up a FIP virtual link

VLAN discovery

The ENode sends a FIP VLAN Request over an available VLAN (for example, port VLAN) to obtain the VLAN running FCoE (FCoE VLAN) from the FCF. The FIP VLAN Request sent must use ALL-FCF-MACs as the destination MAC address.

The FCFs receive the FIP VLAN Request and respond with a unicast FIP VLAN Notification over the same VLAN. The FIP VLAN Notification should contain a list of all FCoE VLANs configured on the FCF. All FIP protocols except VLAN Discovery protocol are performed in the FCoE VLAN.

FCF discovery

The FCF periodically sends unsolicited Discovery Advertisements to multicast MAC address All-ENode-MACs in all its FCoE VLANs. The following table shows the parameters carried in the Discovery Advertisement.

Parameter

Purpose

FCF priority

Used for the ENode to choose an FCF for setting up a VFC link.

Keepalive timer value

Used for the ENode to periodically send VFC link keepalive packets.

FC-MAP

Used for the ENode to generate an FPMA.

FCF MAC address

Used for the ENode to encapsulate and decapsulate frames in FCM.

The ENode can send Discovery Solicitations to multicast MAC address All-FCF-MACs in the FCoE VLANs. In response to a Discovery Solicitation from the ENode, the FCF replies with a solicited Discovery Advertisement to the ENode.

The ENode obtains information about all FCFs in the network through the received Discovery Advertisement. If multiple FCFs respond, the ENode can select a high-priority FCF to set up a VFC link according to the priority carried in the Discovery Advertisements.

VFC link setup

The ENode sends a FIP FLOGI frame or FIP FDISC frame to the selected FCF to set up a VFC link. After receiving the FIP FLOGI frame or FIP FDISC frame, the FCF replies with a FIP ACC frame to allocate an FC ID to the ENode. Then, the ENode combines the FC ID and FC-MAP to generate an FPMA.

Two FCFs use FIP ELP frames to negotiate to set up a VFC link.

VFC link maintenance

After the VFC link is set up, the ENode periodically sends keepalive frames, and the FCF still periodically sends unsolicited Discovery Advertisements. The FCF maintains the VFC link by monitoring the reception of keepalive frames from the connected ENode. The ENode or another FCF connected to the FCF maintains the VFC link by monitoring the reception of unsolicited Discovery Advertisements.

FCoE frame

With a VFC link in place, the ENode and FCF can receive and send packets over VFC interfaces.

FCoE frame format

Figure 4 shows the FCoE frame format.

Figure 4 FCoE frame encapsulated in Ethernet

When the FCF sends a packet, it performs the following operations:

Encapsulates its FCF MAC address as the source MAC address.

Encapsulates the ENode FPMA or peer FCF MAC address as the destination MAC address.

When the ENode sends a packet, it encapsulates its FPMA as the source MAC address and the FCF MAC address as the destination MAC address.

The Priority field indicates the priority of the FCoE frame, defaulting to 3 and configurable.

The VLANID field identifies the VLAN that transports the FCoE frame.

FCoE frame forwarding

Figure 5 FCoE frame forwarding

When forwarded through FCFs, an FCoE frame is decapsulated and encapsulated hop by hop. Figure 5 shows header information encapsulated at each hop.

The FCoE protocol makes sure the packets of one VSAN are transmitted in the same VLAN.

The MAC address of the ENode is a combination of the following:

The FC-MAP as the most significant 24 bits.

The FC ID of the ENode as the least significant 24 bits.

The FCF uses its FCF MAC address to send packets.

DCE lossless transmission

DCE ensures lossless transmission of FCoE frames over Ethernet by using PFC and ETS.

The FCF advertises its FCoE capabilities and the priority used by FCoE frames to the ENode through DCBX. The ENode encapsulates the FCoE frames with that priority.

The FCF reserves the amount of bandwidth (for example, the default 50%) specified by the user for that priority level. If the amount of other traffic is small, FCoE traffic can use more than 50% of the total bandwidth. In the event of large amounts of other traffic, FCoE traffic is guaranteed 50% of the total bandwidth.

When the bandwidth is insufficient for FCoE traffic, the FCF sends a pause frame to notify the upstream device to stop transmitting FCoE frames. When congestion disappears, the FCF notifies the downstream device to continue transmitting.

Application scenario

As shown in Figure 6, the server connects to the FCF through a converged network adapter (CNA). The FCF connects to the LAN and SAN through Ethernet switches and FCFs, respectively. Both FCoE and FC storage devices are supported in the SAN.

Figure 6 FCoE application scenario

新华三官网