FCoE - SAN Protocols Explained

What is FCoE?

Fibre Channel over Ethernet (FCoE) is a computer network technology that encapsulates Fibre Channel frames over Ethernet networks. This allows Fibre Channel to use 10 Gigabit Ethernet networks (or higher speeds) while preserving the Fibre Channel protocol.^[1]

This removes the need to run separate LAN and SAN networks, allowing both networks to be run over a single converged network. In turn, allowing you to keep the latency, security, and traffic management benefits of FC, whilst reducing the number of switches, cables, adapters required within the network - resulting in a reduction to your network TCO.

How does FCoE work?

FCoE works in the same way as native Fibre Channel, the only difference being is that it is encapsulated in an Ethernet header (rather than an FC header) so that it can be sent across the Ethernet network.

With FCoE, both the storage and data traffic is sent out of a shared physical adapter (aka CNA).

Converged Network Adapter (CNA)

A Converged Network Adapter (CNA) is a single network interface card that contains both a Fibre Channel (FC) host bus adapter (HBA) and a TCP/IP Ethernet NIC.

A different VLAN is used for the data and storage traffic, which is then trunked from the CNA up to the switch.

Figure 1 - FCoE CNA.^[2]

Converged Enhanced Ethernet (CEE)

Fibre Channel is a lossless protocol, that uses buffer-to-buffer credits to ensure that frames between the initiator and the target are not lost.

However, Ethernet is not lossless and relies on upper layer protocol of the stack to deal with acknowledgments and retransmissions (i.e TCP). This meant that Ethernet enhancements were needed, and this came in the form of the IEEE Data Center Bridging standards. As Juniper states:

Data center bridging (DCB) is a set of enhancements to the IEEE 802.1 bridge specifications. DCB modifies and extends Ethernet behavior to support I/O convergence in the data center. I/O convergence includes but is not limited to the transport of Ethernet LAN traffic and Fibre Channel (FC) storage area network (SAN) traffic on the same physical Ethernet network infrastructure.^[3]

DCB, also called Converged Enhanced Ethernet (CEE) includes:

Priority Flow Control (PFC)

PFC (IEEE 802.1 Qbb) is a link-level flow control mechanism. Though similar to Ethernet PAUSE (IEEE 802.3x) which stops all traffic on a link for a given period, PFC allows you to split traffic on a link into 8 priorities. Traffic can be independently stopped based on the priority, without having to stop the traffic on any other priority.

Enhanced Transmission Selection (ETS)

ETS (IEEE 802.1 Qaz) provides a mechanism for managing the bandwidth assignment to traffic classes, allowing allocation of bandwidth to the different types of traffic sharing on a converged network.^[4]

Quantized Congestion Notification (QCN)

QCN (IEEE 802.1 Qau) allows for end-to-end congestion control. Congestion is measured at the congestion point, and if congestion is encountered, rate limiting, or back pressure, is imposed at the reaction point to shape traffic and reduce the effects of the congestion on the rest of the network.^[5]

Data Center Bridging Exchange (DCBX)

DCBX is an extension to LLDP, providing the ability to discover the capabilities of DCB peers, detect misconfiguration of peers, and also configure DCB features on peers.

Frame Format

As previously mentioned FCoE is literally just an encapsulation of an FC frame within an Ethernet frame. There is a one-to-one relationship between FC frames and the Ethernet frame. Therefore, FC frames are never segmented and sent across multiple Ethernet frames.

The maximum size of an FCoE frame is 2180bytes. To accommodate this, the network must support an MTU of up to 2.5KB (also known as baby jumbo frames).

Figure 2 - FCoE Frame Format.

Port Types

FCoE provides additional port types to that of standard FC, they are:

• VN_Port - Virtual Node that is instantiated by the ENode; synonymous to the N_Port in Fibre Channel.
• VF_Port - Virtual Fabric port instantiated by the FCF; synonymous to the F_Port in Fibre Channel.
• VE_Port - Virtual Ethernet port instantiated by the FCF; synonymous to the E_Port in Fibre Channel.

Fibre Channel Forwarder

Although FCoE frames are transported over Converged Enhanced Ethernet networks, the forwarding of FCoE frames is not the responsibility of Ethernet forwarding. Instead, FCoE forwarding is the responsibility of a function in the FCoE switch known as the FCoE Forwarder (FCF).^[6]

FCoE Fibre Channel Forwarder (FCF) is a logical Fibre Channel switch within the FCoE switch. FCoE frames are forwarded, by the FCF, based on the FC Destination_ID (D_ID) field within the FCoE frame.

FCoE Initialization Protocol (FIP)

The main goal of FIP is to discover and initialize FCoE capable entities connected to an Ethernet network. FIP is used to not only for initialization functions such as discovering which Fibre Channel entities are available on a layer 2 Ethernet network and the creation of virtual links, but it is also used to verify the state of the virtual links and to destroy virtual links when there is a need to do so.^[7]

Figure 3 - FIP Process.

FIP Snooping

FIP snooping is required in order to prevent man in the middle types of attacks by allowing the lossless Ethernet switches to Dynamically update ACLs and only allow the ENode that performed FIP to transmit frames.^[8]

Useful Links

https://www.emc.com/collateral/hardware/technical-documentation/h6290-fibre-channel-over-ethernet-techbook.pdf

References