Fibre Channel - SAN Protocols Explained

What is Fibre Channel?

FC (Fibre Channel) is a network technology, predominantly used within storage area networks, to provide high-speed, loss-less delivery of raw block data between computer data storage and server devices.

The Fibre Channel network (aka fabric) is a dedicated high-speed, low latency storage network, supporting bandwidth speeds of 2, 4, 6, 8 and 16Gbps, connecting:

• HBAs (Host Bus Adapter) - Dedicated server storage adapters (aka the Initiator)
• Storage Systems - Contains storage controllers, and disks (aka the target).
• FC Switches - High-speed Fibre Channel switches.

The FCP (Fibre Channel Protocol) operates over the FC fabric. Predominantly FCP is the implementation of SCSI over an FC network, or in other words,

SCSI data is encapsulated and transported within FC frames.^[1]

FC Layers

When it comes to learning the various layers of Fibre Channel it is important to understand that FC does not follow the traditional OSI 7 layer model. Instead, it is broken down into 5 layers. As shown below:

FC-0 - Defines the physical media used to link two Fibre Channel ports, including cabling types, optical and electrical parameters for a variety of data rates, maximum transfer distances, and noise limits. Fibre Channel supports two types of cables: Copper and optical.^[2]

FC-1 - Defines the transmission protocol including serial encoding and decoding rules, special characters and error control.^[3]

FC-2 - Defines the transport mechanism of Fibre Channel and the framing rules of the data to be transferred between ports, the different mechanisms for controlling the service of classes and the means of managing the sequence of a data transfer.^[4]

FC-3 - Defines common services required for advanced features such as striping, hunt group, and multicast.

FC-4 - Defines the application interfaces that can execute over Fibre Channel. It specifies the mapping rules of upper layer protocols using the FC levels below.^[5]

Figure 1: FC Layers

FC Frames

Fibre Channel defines a variable length frame consisting of 36 bytes of overhead and up to 2112 bytes of payload for a total maximum size of 2148 bytes. A Start of Frame (SOF) delimiter and End of Frame (EOF) delimiter mark the beginning and end of each Fibre Channel frame.

The CRC is used to detect transmission errors.^[6]

Figure 2: FC Layers.^[7]

Addressing/Naming

World Wide Names

WWN

FC addressing is based upon WWN (World Wide Names). These addresses are unique to each of the FC devices (such as an FC HBA or SAN's) and are 64-bit addresses, consisting of 16 x Hexadecimal values, like so: 15:00:00:f0:8c:08:95:de.

Each device in the SAN is identified by a unique WWN. The WWN contains vendor identifier field, which is defined and maintained by the IEEE.^[8]

The WWN can be compared to a MAC address in the Ethernet world.

WWPN

Whereas the WWN is assigned to the device. The World Wide Port Name's (WWPN) is a unique address assigned to every port of a node (be it HBA or storage system).

WWNN

The World Wide Node Name (WWNN) is assigned to a node in the storage network. For example, the same WWNN can represent multiple network interfaces of a single network node.^[9]

Aliases

Aliases are used to aid in the configuration and troubleshooting, by mapping the World Wide name (from the FC switch or storage system) to a string. For example: the WWPN 15:00:00:f0:8c:08:95:de could be aliased to MAIL-SERVER.

Switch Domain IDs

Within the FC network, each switch is assigned a unique Domain ID. A principle switch is elected within the network which assigns the Domain IDs to the other switches. This Domain ID is then used to form part of the FCID, as per the FLOGI process (described later).

FCNS

The Fibre Channel Name Service (FCNS) is used to exchange the FLOGI database, between switches within the fabric. Allowing each switch to learn where each WWPN is, and how to route traffic there.^[10]

Security

Zoning

Zoning is a security feature that allows the administrator to control which hosts can communicate with each other. This allows the administrator to create zones that prevent servers from communicating with each other over the fabric, ensuring that each server can only communicate to the storage system.

Figure 3: Zoning.^[11]

LUN Masking

LUN masking is used to restrict which LUN is presented to which host or set of hosts. This prevents unauthorized hosts from accessing LUNs and also prevents situations of data corruption that could occur should a host access the wrong LUN.

Login Processes

Fabric Login (FLOGI)

When a device is connected to the fabric, it sends a Fabric Login (FLOGI) request. The switch assigns a 24-bit FCID address, which is equivalent to an IP address.

The FCID is then used by the FC switches to route traffic across the fabric.^[12]

In addition, as part of this process buffer credits are also exchanged with the switch.

Port Login (PLOGI)

Once the FLOGI is complete, the initiator will send the PLOGI request to the switch, in order to inform the Fabric Name Server (FCNS) of its personality and capabilities. Such as:^[13]

• WWNN, WWPN
• Buffer credits for flow control
• clock frequency ('speed capability')
• Upper layer protocol support (eg. SCSI-3, IP)

Process Login (PLRI)

The initiator host will send a PLRI request to the storage target. The storage system will then granted access to the host based on its configured LUN masking.^[14]

Ports/Links

Port Types

There are a number of FC port types. Below outlines the main ones:

Figure 4: FC Ports.

Link Types

• Inter-Switch Link (ISL) - A connection between two switches.
• Trunk - Allows for multiple ISL's to be bundled together as a single link. Traffic is then distributed across each of the members. This is synonymous to port-channels in the world of ethernet.
  
  Figure 5: FC Links.

Classes of Service (CoS)

Applications may require different levels of service and guarantees regarding delivery, connectivity, and bandwidth. Fibre Channel provided different classes of service to accommodate different application needs.^[15]

Flow Control

The goal of flow control is to prevent a situation where a device receives frames faster then it can process. FC provides 2 flow control mechanisms and the mechanism used is dependent upon the service class.

Buffer-to-Buffer (BB_Credits)

Buffer-to-buffer (B2B) is used for Class 3 traffic to provide N_Port to F_Port flow control. Each N_Port begins with a credit pool, that is established during the FLOGI login. Each frame that is sent results in the credit pool is reduced by 1. Once the receiver is ready a R_RDY frame is sent and the sender's credits are replenished.

End-to-End (EE_Credits)

Provides N_Port to N_Port flow control and is used by Class 2 traffic. An initial credit pool is established as part of the PLOGI login. The credit pool is replenished when ACK frames are received from the target.

Useful Links

https://www.iol.unh.edu/testing/storage/fc/tutorial
https://hsi.web.cern.ch/hsi/fcs/spec/overview.htm
https://www.storageinfra.com/fibre-channel-classes-of-service/

References