Description

This multipath failover system was originally written by Linuxcare Inc. for Sun Microsystems.The T3 multipath device driver is a high availability implementation supporting the special fail-over capabilities of the Sun T3 StorEdge enterprise storage units. Moreover when 2 or more T3 units are grouped into a partnered pair this driver provides a reliable communications channel. If a malfunction occurs along one of the paths the system fails-over to an alternate path. This eliminates the single point of failure between the host and the storage devices, bringing an enterprise level of storage operation to the Linux operating system.

  Contact Information

To join one of the mailing lists please send an email to one of the following:

t3-bugs@open-projects.linuxcare.com
t3-dev@open-projects.linuxcare.com

with the word help in the body or subject of the message. Or see the web pages:

http://t3dev.linuxcare.com/mailman/listinfo/t3-bugs

http://t3dev.linuxcare.com/mailman/listinfo/t3-dev

 Installation Requirements

You must have your Host Bus Adapter(s) [HBA] installed properly into your host and your storage devices attached. This driver requires Linux kernel 2.2.x where x is 15 or higher. Older kernels may work but are not certified to operate properly.

The only HBAs tested to work with this software are the QLogic 2100 and QLogic 2200 series Fibre Channel PCI cards. Other HBAs may work, but are currently untested.

  Kernel Installation

Download the kernel patch appropriate for your kernel version. Patch your kernel with it, and reconfigure your kernel to enable the SCSI Configuration -> SCSI Alias Chains option. This will enable the subsystems needed for SUN T3 support.

Recompile your kernel, and boot from it. The following packages will NOT install properly unless the currently running kernel has the SCSI Alias Chains patch

  Installation Instructions for all Linux Distributions

Download the tarball from the web site and unpack it. Run ``make''. Run ``make install''

  Failover Device alias explanation

In order to provide multipath high availability each logical volume created on a T3 unit has 2 communication paths to it and 2 correlating device names under Linux. For example 2 T3 units configured with 3 logical volumes (lets say 1 large volume on the first unit and 2 on the second) would appear as 6 devices within the Linux kernel. Each device has a corresponding alternate path that can be used in the event of a failure on the master path.

     Linux SCSI Devices      FC Controllers        T3 Controller       T3 Volume
        
Vol 1 ---> /dev/sda(u1v1*)-- Qlogic 0 ------------ u1c1 -------------- u1v1
       |   /dev/sdb(u2v1)                        ^ (unit 1)         
       |   /dev/sdc(u2v2)                        | (controller 1)  
       |                                         |
       |                                         v
       --> /dev/sdd(u1v1) -- Qlogic 1 ------------ u2c1 -------------- u2v1
           /dev/sde(u2v1*)                         (unit 2)        |
           /dev/sdf(u2v2*)                         (controller 1)  |-- u2v2

* denotes primary path

In the picture above the devices /dev/sda and /dev/sdd are alternate paths to the same volume, u1v1 (unit 1 volume 1). Under normal operation /dev/sda is used to talk to u1v1 through the path connected by Qlogic 0. To provide HA failover an alias is created to identify /dev/sdd as an alternate path for /dev/sda. In the event that the path for /dev/sda fails (be it a severed cable, a failed controller, etc) /dev/sdd is used to access u1v1 through the u2c1 controller which "fails-over" for u1c1.

IMPORTANT NOTE: Never mount or attempt I/O through both paths at the same time. This would cause the T3 controllers to continuously failover and performance would suffer drastically.

If after a failover has occured (as in the example above) and is then repaired (perhaps the cable was plugged back in), the devices will automatically "failback" restoring performance. It is also possible to manually failback using the t3ctl utility.

 T3 Units Configuration

Complete configuration of your T3 units is beyond the scope of this document. Please refer to the manuals that Sun supplied with your units for that information. However there are several requirements in order to operate properly with this Linux multipath failover system. First, the units must be configured into a partner group, including the fibre channel cable loops on the back of the units. Also the mp_support variable must be set to rw. It is not rw by default. To set this login to your T3 master unit through either telnet or the serial port and use the sys command as such:

     sys mp_support rw

Then run ``sys list'' to verify that it is set properly.

  Failover driver configuration

After you have the multipath failover software installed, the hardware properly connected and configured you are ready to configure the Linux host itself. The first thing to do is to load the qlogic HBA device driver and verify operation by doing the following:

- modprobe qlogicfc

- cat /proc/scsi/scsi and verify that your T3 units have been detected. (there sould be 8 LUNs per T3 unit, with the volumes as type ``Direct-Access'' and the rest as type ``Unknown'')

- t3ctl -l (this will tell you which /dev/sg* devices are used for each /dev/sd* device, the current alias chains, and the online status)

Now you need to add aliases for your alternate paths. Do this with the t3ctl command as such:

          t3ctl -r 60 -t 1000 -s /dev/sgi /dev/sga (This creates an alias chain from /dev/sga(aka /dev/sda) to /dev/sgi(aka /dev/sdd) with the recommended 1000 millisecs/retry and maximum of 60 retires during failover.)

Running t3ctl with the -l option will display the alias information to verify proper alias creation. Set an alias for each alternate path to the volumes that you have created on your T3's. Using the example above, 3 aliases need to be created, one for each primary volume

  Failover Device Operation

When a path to a device fails, the SCSI errors generated by that failure are caught by the alias chain, and the failed commands are retried on the alternate path. If the alternate path succeeds, all future commands to the failed path are automatically redirected to the alternate path, until the primary path returns to online status.