A Fibre Channel Primer: Part 1
W. Curtis Preston SANs certainly qualify as leading-edge technology -- perhaps even bleeding edge. However, Fibre Channel has been around for years. Many people seem to be confused about Fibre Channel as it relates to SANs, which may be why I received so many emails after my column “Why SANs?” (Sys Admin, September 2000). Readers have heard about SANs and the benefits they provide, but the Fibre Channel technology upon which they are based is often a mystery. Perhaps the reason that Fibre Channel is unknown is that, although its been around for a long time, you didn't need to know much about it. For most people, Fibre Channel meant plugging in a Fibre Channel disk to a host, and that was the end of it. They didn't know that they were creating a point-to-point Fibre Channel network, or that they were creating an arbitrated loop when they chained a few storage arrays together on the back of a server. It's not necessary to know a lot about Fibre Channel when using a small network. However, in SAN-land, not knowing Fibre Channel can increase your confusion, frustration, and the cost of the project. This month's column was originally going to cover the different vendors that manufacture and sell SAN equipment, but I decided instead to spend a little time explaining the basics of Fibre Channel and its topologies. The next thing I know, I had enough for two columns! I hope you enjoy part 1 of this Fibre Channel primer. Why Fibre Channel? The purpose of Fibre Channel was to remove the performance barriers of legacy LANs. The performance-enhancing network features of Fibre Channel include:
• Support for other, typically “non-network” protocols, such as SCSI. • Confirmed delivery, thereby enhancing the reliability of the protocol stack. • The option of bypassing the protocol stack for increased performance. • Complete support for traditional network self-discovery. Full support of ARP, RARP, and other self-discovery protocols. • Support for dedicated bandwidth point-to-point circuits, shared bandwidth loop circuits, or scalable bandwidth switched circuits. • True quality of service features, including fractional bandwidth and connection-oriented virtual circuits to guarantee bandwidth for critical backups or other operations. • The option of real circuits or virtual circuits. • Circuit setup time is measured in microseconds using hardware enhanced Fibre Channel protocol. • Extremely low-latency connection and connectionless service. • Automatic self-discovery of all Fibre Channel topologies. • Full support for time-synchronous applications like video, using fractional-bandwidth virtual circuits. • Efficient, high-bandwidth, low-latency transfers using variable length (0-2 KB) frames. Highly effective for protocol frames of less than 100 bytes, as well as bulk data transfer using the maximum frame size. As shown in Table 1, Fibre Channel is the best technology for applications that require high-bandwidth, reliable solutions that scale from small to very large. The Fibre Channel architecture represents a true channel/network integration with an active, intelligent interconnection among devices. A Fibre Channel port only has to manage a simple point-to-point connection. The transmission is isolated from the control protocol, so point-to-point links, arbitrated loops, and switched topologies are used to meet the specific needs of an application. The fabric is self-managing. Nodes do not need station management, which greatly simplifies implementation. Who is Fibre Channel? Unfortunately, there are several groups that are attempting to standardize Fibre Channel. This is one of the concerns that some industry analysts have. Among those trying to standardize Fibre Channel and SANs are the Storage Networking Industry Association (www.snia.org), the Fibre Channel Industry Association (www.fibrechannel.com), the EMC-led Fibre Alliance (www.fibrealliance.org), and the Sun-led Jiro platform expert group (www.jiro.com). There are also at least two independent laboratories for Fibre Channel testing. The Interoperability Laboratory at the University of New Hampshire (www.iol.unh.edu) develops test suites for vendors to check compliance with the Fibre Channel standard. The Computational Science and Engineering Laboratory at the University of Minnesota (www.lcse.umn.edu) is focused on functionality and extending the application of Fibre Channel. I think the SAN and Fibre Channel community is developing like the the UNIX community did. Each of the major vendors seems to want to put their stamp on how SANs operate, thinking that this will somehow result in more hardware and software sales of their platform. However, I see that the platform-independent storage vendors (Brocade, Crossroads, etc.) tend to belong to all of these organizations. Hopefully, the platform-dependent vendors (Sun, HP, EMC) will listen to these voices of reason when discussing interoperability issues. What is Fibre Channel? Although a Fibre Channel network can be as simple as a Fibre Channel disk drive plugged into a Fibre Channel card (or HBA) on the back of a server, it is much more. Fibre Channel is actually a set of standards that defines a multi-layered architecture that transfers data. The layers are numbered FC-0 to FC-4, thus they do not map directly to the seven-layer OSI networking model. FC-4 (the top layer) defines how a Fibre Channel network will communicate with upper-level applications, such as SCSI and IP. (As I will describe, both SCSI and IP traffic can be sent across a Fibre Channel network.) FC-3 is currently under development and will be used by applications requiring more than one port, such as data striping. FC-2 is similar to the Media Access Control (MAC) layer, and defines how data from upper-level applications is split into frames for transport over the lower layers. FC-0 and FC-1 are similar to the physical layer in the OSI model. FC-1 defines how frames are encoded and decoded for transport across those media types, and FC-0 defines the various media types that can carry Fibre Channel data. How does data get from a SCSI tape drive to a SCSI disk via a Fibre Channel network? First, a server requests that the data be retrieved from tape. SCSI and other protocols define how the data is initially retrieved from the tape drive. The SCSI application (as defined by the SCSI protocol) then passes the data to the Fibre Channel network, which receives the data from the SCSI application (as defined in FC-4). It then segments the data into frames (FC-2), encodes the frames (FC-1), and sends them across the various types of media that comprise the physical portion of the network. Depending on the type of network, the Fibre Channel frames may travel over fiber optic or copper cables. They may also travel over one or more switches, hubs, or routers, depending on the type of Fibre Channel topology that is being used. They will eventually arrive at the port to which the disk drive is attached (typically after going into and out of the server's CPU). The frames are then decoded (FC-1), reassembled (FC-2), and given back to the SCSI application for transfer to disk. In this example, SCSI is said to be running on top of Fibre Channel. As far as the server that requested this transfer is concerned, it sent a SCSI request that went directly to the tape drive, which transferred the data via SCSI to its CPU, so that it could transfer it via SCSI to the disk drive. However, in this case, the SCSI commands and data were travelling “on top of” a Fibre Channel network that did the data transfer. This allows the servers to talk SCSI, something they understand very well, while we use Fibre Channel to significantly update how the data is actually transferred. It is similar to the way millions of servers have continued to use IP to talk to each other, while the various lower network levels have been updated to include FDDI, ATM, and Gigabit Ethernet. (In fact, these lower layers now include Fibre Channel, since IP can also travel on top of a Fibre Channel network just like SCSI.) One surprise from the data path described above is that Fibre Channel can run over both fiber and copper media. Fiber can run longer distances than copper, but copper cables (including video cable, miniature cable, and twisted pair) are perfectly viable (and less expensive) alternatives for shorter networks. The most common copper cable is twisted pair and uses a DB-9 connector. The cable lengths can run up to 30 meters, depending on the type of cabling, shielding, and components used, but copper is obviously highly susceptible to EMI (Electro-Magnetic Interference) problems. Fiber optic cables, on the other hand, can run up to 175, 500, and 10,000 meters, depending on the type of cabling used. The first choice to make is between multimode and singlemode fiber. Multimode fiber can carry multiple light rays, or modes, simultaneously by transmitting each mode at a slightly different reflection angle within the fiber core. Since the modes will disperse over longer lengths (referred to as modal dispersion), multimode fiber is used for shorter distances (under 500 M). Singlemode fiber, on the other hand, has a much thinner core that is designed to only transmit one light ray, or mode. This allows it to go much greater distances than multimode fiber, up to 10 KM. With multimode fiber, there are also two core diameters to choose from. The larger, 62.5-micron core, can be used up to a length of 175 M, but the 50-micron core is able to extend up to 500 M. This information is summarized in Table 2. Note the different spelling between the phrases Fibre Channel and Fiber Optics. Originally, Fibre Channel was intended to run only on fiber optic cables. However, once support for copper cabling had been added, the standards committee found themselves in a quandary. People already knew what Fiber/Fibre Channel was, but the committee wanted to somehow show that Fiber/Fibre Channel didn't necessarily mean fiber optic cables. The result was to use the French spelling of fibre, resulting in the current usage of Fibre Channel. Well, at least they tried! Fibre Channel Topologies There are three Fibre Channel network topologies, the simplest and least expensive of which is point-to-point. Slightly more complex and more expensive than point-to-point is arbitrated loop, although this cost comes with much more functionality. The most expensive and complex topology is a Fibre Channel, the fabric topology, although it has even more functionality than an arbitrated loop. (My Sys Admin, January 2001 column will discuss the different Fibre Channel topologies in detail.) Fibre Channel Ports Since the names of the different types of Fibre Channel ports will be used in the explanation of the different Fibre Channel topologies, this is an appropriate time to explain the different types of Fibre Channel ports. There are three “basic” types of ports -- the N_Port, the F_Port, and the E_Port. As you add arbitrated loop capabilities to these basic ports, they take on the “combined” names of NL_Port, FL_Port, and G_Port:
N_Port -- An N_Port is a node port, or a port on a disk or computer. A port that is only an N_Port can only communicate with another N_Port on another node, or to a switch. F_Port -- An F_Port is a fabric port, which is found only on a switch. A port that is only an F_Port can only connect to another N_Port via a point-to-point connection. L_Port -- The term L_Port is not really used in Fibre Channel, but the “L” in the name implies that it is a port that can participate in an arbitrated loop. Typically, this L is added to the end of an N_Port or F_Port to create an NL_Port or FL_Port. NL_Port -- A node port with arbitrated loop capabilities; a port on a node that can connect to another node or a switch (see the N_Port definition), or it can participate in an arbitrated loop (see the L_Port definition). FL_Port -- A fabric port with arbitrated loop capabilities; that is, a port on a switch that can connect to a node (see the F_Port definition) or an arbitrated loop (see the L_Port definition). E_Port -- An E_Port is an expansion port on a switch that is used to connect to other switches via their E_Ports to form a large fabric. G_Port -- A G_Port is a generic port on a switch that can act as an E_Port, an FL_Port, or an F_Port, depending on what connects to it.
Another term to know is HBA, or Host Bus Adaptor. This is the industry term for a Fibre Channel interface card that plugs into a host, such as a PCI or SBUS card. Addressing The IEEE works with each manufacturer of Fibre Channel equipment to assign a fixed, unique, 64-bit address to each port, and is referred to as its WorldWide Name, or WWN. This is similar to the way MAC addresses are assigned to Network Interface Cards (NICs). Then there are two dynamic addresses that will be assigned when this port connects to a Fibre Channel network. If it connects to an arbitrated loop, it will be assigned a dynamic 8-bit address, referred to as its arbitrated loop physical address, or AL_PA. If it connects to a fabric, it will be assigned a dynamic 24-bit address, referred to as its Native Address Identifier. When a port is connected to both an arbitrated loop and a fabric, the lowest eight bits of the Native Address Identifier become the AL_PA. Now that I've defined these terms, I can cover the different Fibre Channel topologies: point-to-point, fabric, and arbitrated loop in the January column. Special thanks to Ayaz Mudarris for his assistance with this month's column.
About the Author
W. Curtis Preston is a principle consultant at Collective Technologies (http://www.colltech.com), and has specialized in designing and implementing enterprise backup systems for Fortune 500 companies for more than 7 years. Portions of this article are excerpted from his O'Reilly book UNIX Backup & Recovery. Curtis may be reached at: curtis@backupcentral.com.
|