Brocade200 E Metro Cluster Design

Notes:	Hide Notes

Slide 1: MetroCluster Design and Implementation Guide Jim Lanson, Network Appliance, Inc. May, 2007 \| TR-3548 Abstract This document is intended to serve as a guide for architecting and deploying MetroCluster in a customer environment. This design and implementation guide describes the basic MetroCluster architecture, considerations for deployment, and best practices. As always, please refer to the latest technical publications on the NOW™ (NetApp on the Web) site for specific updates on processes, Data ONTAP® command syntax, and the latest requirements, issues, and limitations. This document is intended for field personnel who require assistance in architecting and deploying a MetroCluster solution. Slide 2: Table of Contents 1. 1.1. 1.2. 1.3. 2. 2.1. 2.2. 3. 3.1. 3.2. 3.3. 3.4. 4. 4.1. 4.2. 4.3. 4.4. 4.5. 5. 5.1. 5.2. 5.3. 5.4. 6. 6.1. 6.2. 7. 7.1. 7.2. 7.3. 7.4. 7.5. Introduction ..................................................................................................................................................4 Intended Audience .............................................................................................................................4 Scope .................................................................................................................................................4 Requirements and Assumptions ........................................................................................................4 Overview ......................................................................................................................................................5 The Basics..........................................................................................................................................5 Fibre Channel Switch Overview.........................................................................................................9 Deployment Planning.................................................................................................................................11 Data Gathering.................................................................................................................................11 Distance Considerations ..................................................................................................................12 Cluster Configuration Checker.........................................................................................................14 Best Practices and Recommendations ............................................................................................14 Installation and Configuration ....................................................................................................................15 Primary Site......................................................................................................................................15 Remote Site......................................................................................................................................22 Testing..............................................................................................................................................24 Adding More Shelves .......................................................................................................................25 Best Practices and Recommendations ............................................................................................26 Site Failover and Recovery........................................................................................................................28 Site Failover .....................................................................................................................................28 Split Brain Scenario..........................................................................................................................28 Recovery Process ............................................................................................................................28 Giveback Process ............................................................................................................................29 Relevant Documentation ...........................................................................................................................30 MetroCluster Documentation ...........................................................................................................30 Brocade Switch Documentation.......................................................................................................30 Appendixes ................................................................................................................................................31 Appendix A: Switch Ports ................................................................................................................31 Appendix B: Switch Software Upgrade Procedure ..........................................................................32 Appendix C: Fabric MetroCluster Worksheet .................................................................................33 Appendix D: Fabric MetroCluster (16-Port Switches) ......................................................................35 Appendix E: Fabric MetroCluster (8-Port Switches) ........................................................................37 MetroCluster Design & Implementation Guide – version 1.0 2 2 Slide 3: Figures and Tables Figure 2-1) Stretch MetroCluster. ...........................................................................................................................5 Figure 2-2) Fabric MetroCluster. ............................................................................................................................6 Figure 2-3) SyncMirror............................................................................................................................................7 Table 2-1) Hardware disk ownership......................................................................................................................8 Figure 2-4) Dual Fabric MetroCluster. ..................................................................................................................10 Figure 2-5) 16-port switch.....................................................................................................................................10 Figure 2-6) 8-port switch.......................................................................................................................................11 Figure 4- 1) Primary node cabling – appliance to switches..................................................................................19 Figure 4-2) Primary node cabling – disk shelf to switch.......................................................................................20 Figure 4-3) Remote node – appliance to switch...................................................................................................23 Figure 4-4) Remote node cabling – disk shelf to switch.......................................................................................24 Figure 4-5) 16-port switch port assignments. .......................................................................................................25 Figure 4-6) 8-port switch port assignments. .........................................................................................................25 Figure 4-7a) Adding disk shelves to primary node. ..............................................................................................26 Figure 4-7b) Adding disk shelves to remote node................................................................................................26 MetroCluster Design & Implementation Guide – version 1.0 3 3 Slide 4: 1. 1.1. Introduction Intended Audience The information in this document is intended for field personnel and administrators who are responsible for architecting and deploying successful MetroCluster high-availability and disaster-recovery configurations. A brief overview of MetroCluster is presented in order to establish baseline knowledge before discussing implementation planning, installation, configuration, and operation. 1.2. Scope This document covers the following MetroCluster configurations: • • Stretched or non-switched MetroCluster (NetApp storage) Fabric or switched MetroCluster (NetApp Storage) Topics that apply only to non-switched MetroCluster refer to Stretch MetroCluster. Topics that are specific to NetApp storage-based fabric MetroCluster refer to Fabric MetroCluster. Topics that apply to all configurations refer simply to MetroCluster. Other than a short description, V-Series MetroCluster is not covered in this document. This document refers mostly to Fabric MetroCluster, because Fabric MetroCluster configurations introduce an increased level of complexity in design and implementation. To many, Stretch MetroCluster is simply an active-active configuration with longer cables and mirroring. The introduction of Fibre Channel switches and longer distances requires much more consideration and discussion. The operational sections (creating mirrors, forced takeover, etc.) apply to both. 1.3. Requirements and Assumptions For the methods and procedures described in this document to be useful to the reader, the following assumptions are made: • The reader has at least basic Network Appliance administration skills and has administrative access to the storage system via the command-line interface. • The reader has a full understanding of clustering as it applies to the NetApp storage controller environment. • The reader has at least a basic understanding of Fibre Channel switch technology and operation, along with access to the switches via command line. • In the examples in this report, all administrative commands are performed at the storage system or Fibre Channel switch command line. MetroCluster Design & Implementation Guide – version 1.0 4 4 Slide 5: 2. 2.1. Overview The Basics FEATURES MetroCluster configurations consist of a pair of active-active storage controllers configured with mirrored aggregates and extended distance capabilities to create a high-availability disaster recovery solution. The primary benefits include: • • • • Faster high availability and geographic disaster recovery protection Minimal risk of lost data, easier disaster recovery, and reduced system downtime Quicker recovery when disaster occurs Minimal disruption to users and client applications METROCLUSTER TYPES Stretch MetroCluster (sometimes referred to as non-switched) is simply an active-active configuration that can extend up to 500m depending on speed and cable type. It also includes synchronous mirroring (SyncMirror®) and the ability to do a site failover with a single command. See Figure 2-1. Additional resiliency can be provided through the use of multipathing. Total Distance 500m @2Gbps (270m @4Gbps ) Primary NetApp FAS 3020 Secondary NetApp CI FAS 3020 activity status power activity status power DS 14 MK2 AT Pow er Network Appl ianc e NetworkAppliance Net work Applia nce Network Appli anc e NetworkAppliance NetworkApplia nce Network Applia nc e NetworkAppliance NetworkApplia nce NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance NetworkApplia nce NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance Network Appl ianc e NetworkAppliance NetworkAppliance Network Appli anc e Network Appl ianc e DS 14 MK2 AT Pow er F ault Module A F ault Module A Module B Module B Sys tem Shelf ID Shelf ID Sys tem 212A 212A 212A 212A 21 2A 212A 212A 212A 21 2A 212A 212A 212A 212A 212A 13 12 11 10 09 08 07 06 05 04 03 02 01 00 212A 212A 212A 212A 212A 212A 21 2A 212A 212A 212A 212A 212A 212A 212A 13 12 11 10 09 08 07 06 05 04 03 02 01 00 Primary FC Storage DS 14 MK2 AT P ower Secondary FC Storage DS 14 MK2 AT P ower NetworkAppliance Network Appl ianc e NetworkAppliance NetworkAppliance Network Appl ianc e NetworkAppliance NetworkAppliance Network Appli anc e Network Appl ianc e Net work Applia nce Network Applia nc e Network Appl ianc e NetworkAppliance Net work Applia nce NetworkAppliance NetworkAppliance NetworkAppliance NetworkAppliance Network Appl ianc e NetworkAppliance NetworkAppliance Network Appl ianc e NetworkAppliance Net work Applia nce Network Appli anc e NetworkAppliance Net work Applia nce Network Applia nc e Fau lt Fau lt Module A Module A Module B Module B S ystem S ystem Shelf ID Shelf ID 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 21 2A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 13 12 11 10 09 08 07 06 05 04 03 02 01 00 13 12 11 10 09 08 07 06 05 04 03 02 01 00 Secondary FC Mirror Primary FC Mirror Figure 2-1) Stretch MetroCluster. Fabric MetroCluster (also referred to as switched) uses four Fibre Channel switches in a dual fabric configuration and a separate cluster interconnect card to achieve an even greater distance (up to 100km) between primary and disaster recovery locations. See Figure 2-2. MetroCluster Design & Implementation Guide – version 1.0 5 5 Slide 6: Fabric (switched) MetroCluster Total Distance 100Km Primary Node NetApp FAS 3020 Secondary Node NetApp FAS 3020 activity status po r we activity status power CI CI Brocade 200E ISL ISL CI CI Brocade 200E DS14 M K2 NetworkA pplia nce Net workAppliance Net workAppliance Net workAppliance NetworkA ppliance NetworkA ppliance NetworkA ppliance Network Appliance Network Appliance Ne tworkApplianc e Ne tworkApplianc e Ne tworkApplianc e NetworkA ppliance NetworkA ppliance AT P ow er Netw orkAppliance N etworkAppliance N etworkAppliance N etworkAppliance Netw orkAppliance Netw orkAppliance Brocade 200E Netw orkAppliance Ne tworkAppliance Ne tworkAppliance NetworkA ppliance NetworkA ppliance NetworkA ppliance Net workAppliance DS 14 MK2 Net workAppliance AT Power Fault Module A Fault Module A Module B S ystem Module B Sys tem Shelf ID Shelf ID 212A 212A 212A 21 2A 21 2A 21 2A 21 2A 212A 212A 212A 212A 212A 212A 212A 21 2A 21 2A 21 2A 212A 212A 212A 212A 212A 212A 21 2A 21 2A 21 2A 212A 212A 13 12 11 10 09 08 07 06 05 04 03 02 01 00 13 12 11 10 09 08 07 06 05 04 03 02 01 00 Primary FC Storage DS14 MK2 Netw ork Appliance Ne tworkAppliance Ne tworkAppliance Ne tworkAppliance Network Appliance Network Appliance Network Appliance Netw orkAppliance Netw orkAppliance N etworkAppliance N etworkAppliance N etworkAppliance Netw orkAppliance Netw orkAppliance Secondary FC Storage DS14 MK2 N etworkA ppl iance N etworkA ppl iance Netw orkAppliance N etworkA ppliance N etworkA ppliance Net workAppliance N etworkA ppliance Ne tworkAppliance Netw orkApplianc e Netw orkApplianc e N etworkA pplia nce Netw orkAppliance Netw orkAppliance N etworkA ppl iance AT P ow er AT P ower Fault Fault Module A Module A Module B Module B System System Shelf ID Shelf ID 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 212A 13 12 11 10 09 08 07 06 05 04 03 02 01 00 212A 21 2A 212A 212A 21 2A 212A 212A 212A 212A 212A 212A 212A 212A 212A 13 12 11 10 09 08 07 06 05 04 03 02 01 00 Secondary FC Mirror Primary FC Mirror Figure 2-2) Fabric MetroCluster. V-Series MetroCluster is simply either of the above configurations with a NetApp V-Series system. Because of the architectural differences between V-Series and a standard active-active configuration, V-Series MetroCluster has additional flexibility when it comes to the maximum number of spindles and the Fibre Channel switches. See the V-Series Compatibility Guide on the NOW site for additional information. The implementation of V-Series MetroCluster is outside the scope of this document. See the V-Series documentation on the NOW site. For more information about MetroCluster, see the Data ONTAP Cluster Installation and Administration Guide (for Data ONTAP 7.1) or the Data ONTAP Active-Active Configuration Guide (for Data ONTAP 7.2 or later). (For Data ONTAP 7.0 and earlier, see the Data Protection Online Backup and Recovery Guide and the NetApp Hardware Cluster Guide.) Note: When referring to the documentation just listed, use the wiring diagrams in this technical report. The diagrams in this document facilitate easier installation and expansion of MetroCluster. MIRRORING NetApp SyncMirror, an integral part of MetroCluster, combines the disk-mirroring protection of RAID1 with NetApp industry-leading RAID4 and RAID-DP™ technology. In the event of an outage—whether it's due to a disk problem, cable break, or host bus adapter (HBA) failure—SyncMirror can instantly access the mirrored data without any operator intervention or disruption to client applications. SyncMirror maintains a strict physical separation between the two copies of your mirrored data. Each of these copies is referred to as a plex. Each controller’s data has its “mirror” at the other site. MetroCluster Design & Implementation Guide – version 1.0 6 6 Slide 7: Aggregate Located at Partner Site Mirror Plex0 Plex1 Pool0 Pool1 Figure 2-3) SyncMirror Pool s and Plexes. When SyncMirror is licensed and hardware ownership is used (discussed in "Disk Ownership"), spare disks are split into two pools—Pool0 and Pool1. Each plex of a mirror uses disks from separate pools. Disks assigned to Pool0 must be on separate loops from disks assigned to Pool1. When software ownership is used, disks are explicitly assigned by the administrator, as described in “Disk Ownership.” To maximize availability, Pool0 and Pool1 disks need to be on separate loops and use separate HBAs, cables, and shelves. Before enabling the SyncMirror license, ensure that disks for each pool are located on the appropriate loops that are fault isolated from each other. DISK OWNERSHIP In a MetroCluster where disk shelves at either side are mirrored and thus accessible by either controller, disk ownership comes into play. There are two methods of establishing disk ownership: hardware and software. Hardware-based ownership is the default for all platforms except V-Series, FAS 6000 series, and FAS 3040/3070. Software disk ownership capability became available in Data ONTAP 6.3.1. A brief description of each method follows. For more detail, see "Installation and Configuration," later in this document or the Data ONTAP documentation. Hardware disk ownership establishes which controller owns which disks by how the shelves are connected. For more information, see the System Configuration Guide: http://now.netapp.com/NOW/knowledge/docs/hardware/NetApp/syscfg/ MetroCluster Design & Implementation Guide – version 1.0 7 7 Slide 8: Model FAS9xx Pool Ownership Slots 2, 3, 4, 5, and 7 are Pool0 Slots 8, 9, and 10 are Pool1 Optional software-based ownership and pool selection (as of Data ONTAP 7.1.1) FAS3020/ 3050 0a, 0b, and slots 1 and 2 are Pool0 (slot 1 is usually NVRAM) 0c, 0d and slots 3 and 4 are Pool1 Optional software-based ownership and pool selection (as of Data ONTAP 7.1.1) FAS3040//3070 FAS60xx V-Series Software-based ownership and pool selection Software-based ownership and pool selection Software-based ownership and pool selection Table 2-1) Hardware disk ownership. Note: Physical connectivity is extremely important to ensure adequate distribution of disks between the two pools. Software disk ownership allows greater flexibility in cabling by allowing disk ownership to be assigned through explicit commands. Rather than determine which controller owns the disk by hardware connection, the ownership information is written on each disk. Because disk ownership can be implemented by a system administrator, it is important to perform the configuration to maximize availability. For more information, see "Installation and Configuration," later in this document. FIBRE CHANNEL SAN IN A METROCLUSTER WORLD For those who are experienced in Fibre Channel technology, and storage area networks (SANs) in particular, there are some differences and restrictions worth discussing relative to how Fabric MetroCluster utilizes this technology. Fabric MetroCluster configurations use Fibre Channel switches as the means to separate the controllers by a greater distance. The switches are connected between the controller heads and the disk shelves, and to each other. Each disk spindle or LUN individually logs into a Fibre Channel fabric. Except for the V-Series, the nature of this architecture requires, for performance reasons, that the two fabrics be completely dedicated to Fabric MetroCluster. Extensive testing was performed to ensure adequate performance with switches included in a Fabric MetroCluster configuration. For this reason, Fabric MetroCluster requirements prohibit the use of any other model or vendor of Fibre Channel switch than the Brocade 200E included with the Fabric MetroCluster. Also for performance reasons there is a current maximum of 336 spindles. Higher spindle count solutions will be available in the future. Keep in mind that this is for Fabric MetroCluster only. The maximum number of disks in a Stretch MetroCluster depends solely on the NetApp model. In a traditional SAN, there is great flexibility in connecting devices to ports as long as the ports are configured correctly and any zoning requirements are met. In a MetroCluster, when using hardware ownership, Data ONTAP expects certain devices to be connected to specific ports or ranges of ports. It is therefore critical that cabling be exactly as described in the installation procedures. Also, no switchspecific functions such as trunking or zoning are currently used in a NetApp Fabric MetroCluster (non-VSeries), making switch management minimal. RESTRICTIONS For further restrictions in using a Fabric MetroCluster configuration, see the Data ONTAP Cluster Installation and Administration Guide (for Data ONTAP version 7.1) or the Data ONTAP Active-Active MetroCluster Design & Implementation Guide – version 1.0 8 8 Slide 9: Configuration Guide (for Data ONTAP version 7.2 or later). (For Data ONTAP 7.0 and earlier, see the Data Protection Online Backup and Recovery Guide and the NetApp Hardware Cluster Guide.) COMPONENTS A NetApp fabric includes the following components: • • A clustered pair of FAS900, 3000, or 6000 series controllers running Data ONTAP 6.4.1 or later (See the MetroCluster Compatibility Matrix on the NOW site for supported models.) Four Brocade Fibre Channel switches with supported firmware (See the MetroCluster Compatibility Matrix on the NOW site for supported models.) There is a pair at each location. Supported models may differ between locations but must be the same at each given location. Brocade Extended Distance license (if over 10km) Brocade Full-Fabric License Brocade Ports-On-Demand (POD) licenses for second 8 ports (if using a 16-port Brocade 200E switch) A VI-MC cluster adapter A syncmirror_local license A cluster_remote license A cluster license • • • • • • • METROCLUSTER VERSUS STANDARD SYNCHRONOUS REPLICATION As a disaster recovery solution, MetroCluster is often compared with other synchronous replication products. Even though it includes SyncMirror, MetroCluster can be differentiated by the following features: • • • Low-level RAID mirroring (less performance impact) Site failover with a single command No extensive scripting required to make data available after failover 2.2. Fibre Channel Switch Overview OPERATION OVERVIEW The NetApp Fabric or switched MetroCluster configuration uses four Brocade 200E switches in a dual fabric configuration to connect two active-active controllers. These switches cannot be combined with any other switch model. SWITCH FABRIC CONFIGURATION A Fabric MetroCluster contains two switch fabrics. A switch fabric consists of a switch on the primary controller connected to a switch on the remote controller (see Figure 2-1). The two switches are connected to each other through ISL (Inter-Switch Link) cables. MetroCluster Design & Implementation Guide – version 1.0 9 9 Slide 10: Fabric 1 SW1 SW3 Fabric 2 SW2 SW4 Figure 2-4) Dual Fabric MetroCluster. Figure 2-4 shows a Fabric MetroCluster configuration in which the first fabric begins at “SWITCH 1” on the primary side and is completed by connecting the ISL cable to the first switch, “SWITCH 3“, on the remote side. The second fabric is created by using “SWITCH 2” on the primary side, connected through a second ISL cable to the second switch “SWITCH 4” on the remote side. The reason for two fabrics is redundancy. The loss of a switch in a fabric or the loss of a fabric will not affect the availability of the Fabric MetroCluster. Note: Only one ISL is supported per fabric. SWITCH PORT ALLOCATION For purposes of this discussion, hardware disk ownership is assumed. The Fibre Channel switch ports are divided into banks and pools. Each switch has two banks, dividing the switch into two equal parts. (See Figures 2-5 and 2-6.) A storage controller connects to a bank through its Fibre Channel ports (HBA or embedded). It then owns the disks that are connected to ports of the opposite bank. If the storage system connects to bank 1, only shelves connected to bank 2 belong to it. Additionally, Pool0 disks are considered “local” – all reads will be satisfied from those disks only. Pool1 disks are “remote”; they will be written to, but cannot be read from under normal circumstances. Together, these two rules limit the usable ports for a certain disk pool to 4 on a 16-port switch, and to 2 on an 8-port switch. 16-PORT SWITCHES As shown in Figure 2-5, ports 0-3 and 8-11 belong to Pool0, while ports 4-7 and 12-15 belong to Pool1. The Brocade 200E 16-port switch has ports 0-15 licensed and operational, as shown in Figure 2-5. Note: Pools are identified only as either Pool0 or Pool1. However, Data ONTAP can assign arbitrary numbers to plexes. 16 Port Switch Pool 0 1 Figure 2-5) 16-port switch. 8-PORT SWITCHES The 8-port Brocade 200E switch physically has 16 ports and looks like the 16-port version. However, only ports 0-7 are licensed, have small form-factor Pluggables (SFPs), and are operational, as shown in Figure 2-6. MetroCluster Design & Implementation Guide – version 1.0 10 10 Slide 11: Note: Do not install SFPs in unused ports 8-15. This might cause Data ONTAP to operate as if it's a 16port switch and erroneously assign banks, quadrants, and pools to unused ports. As shown in Figure 2-6, ports 0-1 and 4-5 belong to Pool0, while ports 2-3 and 6-7 belong to Pool1. 8 Port Switch Pool 0 Pool 1 Figure 2-6) 8-port switch. LICENSES A number of licenses are required for the Fibre Channel switches. When ordered with a Fabric MetroCluster configuration, the switches should include all necessary licenses. For reference, they are: • • • Full Fabric License Extended Distance License (if over 10km) Brocade Ports-On-Demand (POD) licenses for second 8 ports (if using a 16-port Brocade 200E switch) 3. 3.1. Deployment Planning Data Gathering CUSTOMER REQUIREMENTS To facilitate a successful MetroCluster installation, the following information should be gathered early on. ENVIRONMENTAL Distance between the primary and remote sites. This information is necessary to determine which type of MetroCluster is appropriate, or even whether either version is appropriate. In calculating the effective distance, factors such as cable type, speed, and number of patch panels must be considered. (See Section 3.2, "Distance Considerations.") Although NetApp recommends that dedicated dark fiber be used for a MetroCluster, WDM devices are supported. Refer to the Brocade Compatibility Guide at www.brocade.com for supported devices. SETUP Gather the following items before you begin any MetroCluster installation: NetApp and Fibre Channel switch licenses Hostnames and IP addresses for each of the nodes and the Fibre Channel switches Brocade switch licenses Cables MetroCluster Design & Implementation Guide – version 1.0 11 11 Slide 12: 3.2. Distance Considerations OVERVIEW Stretch MetroCluster can support a maximum of 500 meters between nodes at a speed of 2Gbps. Fabric MetroCluster, through the use of Fibre Channel switches, extends this distance to 100km at the same speed. At 4Gbps speeds, these distances are roughly cut in half. This extended distance capability gives customer greater flexibility in the physical location of the active-active controllers while maintaining the high-availability benefits of clustered failover. This section describes a number of factors that affect the overall effective distance permissible between the MetroCluster nodes. • • • • Physical distance Number of connections Desired speed Cable type PHYSICAL DISTANCE As stated earlier, the Stretch MetroCluster configuration can extend to a maximum of 500m (2Gbps). This distance is reduced by speed, cable type, and number of connections. A Fabric MetroCluster can extend out to 100km. This distance is affected by the same factors. At a distance of 100km, latency would be around 1ms. Greater distances would obviously result in greater latencies (500km = 5ms), which may be unacceptable to an application. CABLE TYPE As shown in Table 3.1, the cable type affects both distance and speed. Single-mode cable is supported only for the Inter-Switch Links. Example 1: A customer has 250 meters between sites and wants to run at 4Gbps. The OM-3 cable type is required. Example 2: A customer currently has a MetroCluster configuration running at 2Gbps with a distance of 300 meters over OM2 cabling and wants to upgrade to 4Gbps speeds. Upgrading the cabling will not help, because OM3 has a maximum of 270 meters. In this case the choices would be: • • Remain at 2Gbps speeds. Customers with the new ESH4 disk shelves could still use them at this distance, as long as the shelf speed is set to 2Gbps. Test current optical network infrastructure to make sure that attenuation and latency are acceptable. Fibre Type OM-2 (50/125UM) Data Rate 1 Gb/s 2 Gb/s 4Gb/s 1 Gb/s 2 Gb/s 4Gb/s 2 Gb/s 4Gb/s Max Distance (M) 500 300 150 860 500 270 10,000* 10,000* OM-3 (50/125UM) OS1 Single Mode (9/125UM) Table 3-1) Cables, speed, and distance. MetroCluster Design & Implementation Guide – version 1.0 12 12 Slide 13: The maximum distance shown here are is typically due to the standard 1310nm SFPs. Use of highpower SFPs can extend this dark fiber up to 30km. Using 1550nm high-power SFPs, a distance of 70100 km can be achieved. This topic is discussed in much greater technical detail in the following technical reports: • • MetroCluster Upgrade Planning Guide (TR3517) Optical Network Installation Guide (TR3552) CABINETS Although it is not mandatory for the equipment to be installed as shown in Figure 3-1, this is a physically desirable configuration that facilitates ease of cabling and scalability for expansion. For ease of management and troubleshooting, it is helpful to set shelf IDs based on location. For example, all disk shelves on the primary controller are set to odd numbers (1, 3, 5) while shelves on the remote controller are set to even numbers. This makes identification and location of shelves much easier when using Data ONTAP utilities. NODE A SW1 SW2 42 42 41 41 NODE B SW3 SW4 42 42 41 41 40 40 40 40 39 39 39 39 38 38 38 38 37 37 37 37 36 36 36 36 35 ID 5 35 35 35 34 34 34 34 33 33 33 33 32 32 32 32 31 31 31 31 30 30 30 30 29 ID 3 29 29 29 28 28 28 28 Node A Shelves 27 27 26 26 25 25 Node B Shelves 27 27 26 26 25 25 24 Ne t o r A p p i n c e Ne t o r Ap p i n c e Ne t o r Ap p a n c e e t o r Ap p i n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e wk a l wk a l wk i l N wk a l w k i l Nwk i l wk i l D 14 S Ne t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk l i wk a l Nwk i l wk i l Nwk i l wk l i w k a l M2 K 24 24 Ne t o r Ap p a n c e Ne t o r Ap p i n c e Ne t o r Ap p l n c e e t o r Ap p i n c e Ne t o r Ap p a n c e e t o r A p p a n c e Ne t o r Ap p a n c e wk i l wk a l wk a i Nwk a l wk i l Nwk i l wk i l D 14 S Ne t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk l i wk a l Nwk i l wk a l N wk i l wk i l wk a l M2 K 24 23 ID 1 72F 72 F 72 F FC Po e wr Faut l 23 23 LoopA LoopB Sy m e t s ID 1 72 F 72 F 72 F FC Po e wr Faut l 23 LoopA LoopB Sy m e t s Sh e l f D I Sh e l f D I 22 72 F 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 22 22 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7F 2 22 21 21 21 21 20 FAS3 0 5 0 20 20 FAS3 0 5 0 20 19 act i vy it st t s au p o we r 19 19 act i vy it stts au p o we r 19 18 18 18 18 Node B Mirror 17 17 16 16 15 Ne t o r A p p i n c e Ne t o r Ap p i n c e Ne t o r Ap p a n c e e t o r Ap p i n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e wk a l wk a l wk i l N wk a l w k i l Nwk i l wk i l D 14 S Ne t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk l i wk a l Nwk a l wk l i Nwk i l wk i l w k a l M2 K 15 Node A Mirror 17 17 16 16 15 Ne t o r Ap p a n c e Ne t o r Ap p i n c e Ne t o r Ap p l n c e e t o r Ap p i n c e Ne t o r Ap p a n c e e t o r A p p a n c e Ne t o r Ap p a n c e wk i l wk a l wk a i Nwk a l wk i l Nwk i l wk i l D 14 S Ne t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e Ne t o r Ap p i n c e wk l i wk a l Nwk i l wk i l N wk i l wk a l wk a l M2 K 15 14 ID 2 72 F 72F 72 F FC Pwe or Faut l 14 14 LoopA LoopB Ss m e t y ID 2 7F 2 72 F 72 F FC Pw e or Faut l 14 LoopA LoopB Ss m e t y Sh e l f D I Sh e l f D I 13 72 F 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 13 13 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 13 12 12 12 12 11 11 11 11 10 10 10 10 9 9 9 9 8 ID 9 8 8 8 7 7 7 7 6 6 6 6 5 5 5 5 4 4 4 4 3 3 3 3 2 ID 11 2 2 2 1 1 1 1 Figure 3-1a) Typical Fabric MetroCluster rack installation (minimal). MetroCluster Design & Implementation Guide – version 1.0 13 13 Slide 14: NODE A SW1 SW2 42 42 41 41 NODE B SW3 SW4 42 42 41 41 40 40 40 40 39 39 39 39 38 38 38 38 37 37 37 37 36 Ne t o r Ap p i n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p i n c e wk a l wk i l wk i l Nwk i l wk a l Nw k a l wk a l D 14 S 36 36 Ne t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r A p p i n c e wk i l wk i l wk i l Nwk i l wk a l Nw k i l wk a l M K2 D 14 S Ne t o r A p p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk a l Nwk i l wk i l Nwk l i wk i l w k a l 36 M K2 35 ID 5 72 F 72 F 72 F Ne t o r Ap p a n c e Ne t o r A p p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk i l Nwk i l wk i l Nwk i l wk l i w k a l FC Po e wr Faul t 35 35 Loop A Loop B ID 5 72 F 72 F 7F 2 FC Po e wr Faut l 35 LoopA LoopB Sy m e t s Sy m e t s Sh e f D I l Sh e f D I l 34 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 2F 34 34 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 34 33 33 33 33 32 32 32 32 31 31 31 31 30 Ne t o r Ap p i n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p i n c e wk a l wk i l wk i l Nwk i l wk a l Nw k a l wk a l D 14 S Ne t o r Ap p a n c e Ne t o r A p p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk i l Nwk i l wk i l Nwk i l wk l i w k a l M K2 30 30 Ne t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r A p p i n c e wk i l wk i l wk i l Nwk i l wk a l Nw k i l wk a l D 14 S Ne t o r A p p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk a l Nwk i l wk i l Nwk l i wk i l w k a l M K2 30 FC FC 29 ID 3 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 2F Po e wr Faul t 29 29 Loop A Loop B Sy m e t s Sh e f D I l ID 3 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7F 2 72 F 72 F 72 F 72 F Po e wr Faut l 29 LoopA LoopB Sy m e t s Sh e f D I l 28 28 28 28 Node A Shelves 27 27 26 26 25 25 Node B Shelves 27 27 26 26 25 25 24 Ne t o r Ap p i n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p i n c e wk a l wk i l wk i l Nwk i l wk a l Nw k a l wk a l D 14 S M K2 24 24 Ne t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r A p p i n c e wk i l wk i l wk i l Nwk i l wk a l Nw k i l wk a l D 14 S Ne t o r A p p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk a l Nwk i l wk i l Nwk l i wk i l w k a l M K2 24 23 ID 1 72 F 72 F 72 F Ne t o r Ap p a n c e Ne t o r A p p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk i l Nwk i l wk i l Nwk i l wk l i w k a l FC Po e wr Faul t 23 23 Loop A Loop B Sy m e t s ID 1 72 F 72 F 7F 2 FC Po e wr Faut l 23 LoopA LoopB Sy m e t s Sh e f D I l Sh e f D I l 22 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 2F 22 22 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 22 21 21 21 21 20 FAS3 0 5 0 20 20 FAS3 0 5 0 20 19 act i vy it st t s au p o we r 19 19 act i vy it stts au p o we r 19 18 18 18 18 Node B Mirror 17 17 16 16 15 Ne t o r Ap p i n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p i n c e wk a l wk i l wk i l Nwk i l wk a l Nw k a l wk a l DM1 4 S K2 15 Node A Mirror 17 17 16 16 15 Ne t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r A p p i n c e wk i l wk i l wk i l Nwk i l wk a l Nw k i l wk a l DM1 4 S K2 Ne t o r A p p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk a l Nwk i l wk i l Nwk l i wk i l w k a l 15 14 ID 2 72 F 72 F 72 F Ne t o r Ap p a n c e Ne t o r A p p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk i l Nwk i l wk i l Nwk i l wk l i w k a l FC Po e wr Faul t 14 14 Loop A Loop B Sy m e t s ID 2 72 F 72 F 7F 2 FC Po e wr Faut l 14 LoopA LoopB Sy m e t s Sh e f D I l Sh e f D I l 13 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 2F 13 13 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 13 12 12 12 12 11 11 11 11 10 10 10 10 9 Ne t o r Ap p i n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p i n c e wk a l wk i l wk i l Nwk i l wk a l Nw k a l wk a l DM1 4 S K2 9 9 Ne t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r A p p i n c e wk i l wk i l wk i l Nwk i l wk a l Nw k i l wk a l DM1 4 S K2 Ne t o r A p p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk a l Nwk i l wk i l Nwk l i wk i l w k a l 9 8 ID 4 72 F 72 F 72 F Ne t o r Ap p a n c e Ne t o r A p p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk i l Nwk i l wk i l Nwk i l wk l i w k a l FC Po e wr Faut l 8 8 Loop A Loop B ID 4 72 F 72 F 7F 2 FC Po e wr Faut l 8 LoopA LoopB Sy m e t s Sy m e t s Sh e f D I l Sh e f D I l 7 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 2F 7 7 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 6 6 6 6 5 5 5 5 4 4 4 4 3 Ne t o r Ap p i n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p i n c e e t o r Ap p i n c e Ne t o r Ap p i n c e wk a l wk i l wk i l Nwk i l wk a l Nw k a l wk a l DM1 4 S K2 3 3 Ne t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r A p p i n c e wk i l wk i l wk i l Nwk i l wk a l Nw k i l wk a l DM1 4 S K2 Ne t o r A p p a n c e Ne t o r Ap p i n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk a l Nwk i l wk i l Nwk l i wk i l w k a l 3 2 ID 6 72 F 72 F 72 F Ne t o r Ap p a n c e Ne t o r A p p a n c e e t o r Ap p a n c e Ne t o r Ap p a n c e e t o r Ap p a n c e N e t o r Ap p a n c e Ne t o r Ap p i n c e wk i l wk i l Nwk i l wk i l Nwk i l wk l i w k a l FC Po e wr Faut l 2 2 Loop A Loop B ID 6 72 F 72 F 7F 2 FC Po e wr Faut l 2 LoopA LoopB Sy m e t s Sy m e t s Sh e f D I l Sh e f D I l 1 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 7 2F 1 1 7 2F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 72 F 1 Figure 3-1b) Typical Fabric MetroCluster rack installation (shown half populated). 3.3. Cluster Configuration Checker The Cluster Configuration Checker is a Perl script that detects errors in the configuration of a pair of active-active NetApp controllers. It can be run as a command from a UNIX® shell or Windows™ prompt, but also doubles as a CGI script that can be executed by a UNIX Web server. The script uses rsh or ssh to communicate with the controllers you're checking, so you must have the appropriate permissions for rsh to run on both controllers in the cluster pair. This script detects and reports the following: • • • • • Services licensed not identical on partner (some services may be unavailable on takeover) Options settings not identical on partner (some options may be changed on takeover) Network interfaces configured incorrectly (clients will disconnect during takeover) FCP cfmode settings not identical on controllers with FCP licensed Checks /etc/rc on each controller to see that all interfaces have a failover set This script is available on the NOW site for download. NetApp recommends that it be run as part of the implementation process. If the controllers being implemented were part of an active-active configuration, then the configurations are probably compatible. It never hurts to run this utility anyway, just to be certain. 3.4. Best Practices and Recommendations NetApp strongly recommends that the installation planning worksheets be complete prior to beginning the installation. A little time spent up front will expedite the installation process. MetroCluster Design & Implementation Guide – version 1.0 14 14 Slide 15: 4. Installation and Configuration As in most successful installations, the key is planning and preparation. The collected information outlined in Chapter 3 is essential in completing the installation. The steps of a MetroCluster installation can be divided according to primary or remote site. This chapter outlines these steps. Refer to the Active-Active Configuration Guide for supplementary information. 4.1. Primary Site VERIFY LICENSES Missing licenses take time to obtain, so the first step should be to verify not only that the licenses are available but that they are installed. Follow the steps under "Storage Controller" and "Fibre Channel Switches" to validate the licenses. STORAGE CONTROLLER COMMAND telnet controllername (or IPAddr) Login root “password” License RESULT Login prompt Prompt for password Command line prompt Look for the following to be licensed: Cluster cluster_remote syncmirror _local If any of the above are not licensed, perform the following step. License add “license key” FIBRE CHANNEL SWITCHES If this is a brand new switch, the IP address may not be set. The default address is 10.77.77.77. The default user/password is admin/password. If network connectivity is not yet available, then any switch setup must be performed using the console port on the switch itself. Refer to the Brocade documentation for more information. COMMAND telnet “switch IPAddress” Login admin “password” licenseshow RESULT Login prompt Prompt for password Command line prompt Look for the following to be licensed: brcd200e_whql01:admin> licenseshow Web license Zoning license Fabric Watch license Following two licenses only if 16-port switch: MetroCluster Design & Implementation Guide – version 1.0 15 15 Slide 16: Ports on Demand license - additional 4-port upgrade Ports on Demand license - additional 4-port upgrade Fabric license Extended Fabric license (Only if distance between nodes is greater than 10km.) If any of the above are not licensed, perform the following step. Licenseadd “license key” SWITCH CONFIGURATION PROCEDURE To configure a Brocade 200E switch for a fabric- MetroCluster configuration, complete the following steps. Note: This procedure must be performed on each switch in the MetroCluster configuration. Command telnet “switch IPAddress” Login admin “password” version Result Login prompt Prompt for password Command line prompt The currently installed switch firmware is displayed. Check the MetroCluster Fabric Switch download page on http://now.netapp.com for the currently supported version of the switch firmware. Note: To access this page, go to the Fibre Channel switch link on the Download Software page and select Brocade from the platform list. If your switch firmware is not the supported version, complete the steps outlined in Section 7.2, “Appendix B: Switch Software Upgrade Procedure.” . Download the switch firmware from http://now.netapp.com and install it, as described in the appropriate Brocade switch hardware reference manual. reboot Switchdisable cfgclear cfgdisable cfgsave configdefault configure Configure the switch with default settings. Set the switch parameters. You should set only the following parameters. Fabric parameters = y domain_id = XXX As a best practice, set the domain ID according to the switch number in the configuration. For example, at the primary site, 16 Reboot the switch. Disable the switch. Clear any preexisting configuration MetroCluster Design & Implementation Guide – version 1.0 16 Slide 17: switches 1 and 2 would have domain IDs SW1 and SW2 and switches 3 and 4 at the remote would be SW3 and SW4, respectively. Disable device probing = 1 Virtual Channel parameters (yes, y, no, n): [no] F-Port login parameters (yes, y, no, n): [no] Zoning Operation parameters (yes, y, no, n): [no] RSCN Transmission Mode (yes, y, no, n): [no] Arbitrated Loop parameters (yes, y, no, n): [no] y Send FAN frames?: (0..1) [1] 0 Enable CLOSE on OPEN received?: (0..4) [4] ctrl-d fastboot switchdisable portCfgLPort <port#>,0,0,1 Y 0 to disable FAN frames. Exit the configuration utility. Reboot the switch so that the default settings take effect. Log in to the switch and disable it. Set all ports attached to disk loops to half duplex. Note: Perform this command only for the ports to which disks are attached. Refer to the switch-to-disk tables in “Cable Primary Node” for the port numbers. ON is displayed in the Locked Loop HD field. Configure the Inter-Switch Link (ISL) port. ( 0 disables trunking.) Trunking must be disabled prior to configuring the long-distance ISL port. Configure the long-distance ISL port for an ISL length of up to 10 km. For information about configuring an ISL port for an ISL length greater than 10 km, contact Brocade technical support: http://www.brocade.com/support/contacts.jsp For additional compatibility information, see the Brocade Fabric Aware Compatibility Matrix: http://www.brocade.com/products/interop_and_compatibility.jsp portCfgLPort portcfgtrunkport <ISL_port#> 0 portCfgLongDistance <ISLport#>, "LE" 13 portCfgISLMode <ISLport#>, mode Enable or disable ISL R_RDY mode on a port. Toggling this mode may sometimes be necessary (high error rate on port) when connecting to WDM devices. Enable the switch. Set the switch name. switchenable switchname “Loc-SW1” MetroCluster Design & Implementation Guide – version 1.0 17 17 Slide 18: Loc_SW1 is the name you gave to the current switch. Use a name that is meaningful, such as location (Loc for Local, Rem for Remote) and the switch number. Portname <number>,”name” Best practice: To assist in documenting the installation, names can be assigned to each port. Examples: Portname 12,”ISL” Using the portshow command, this identifies port 12 as being an ISL port. Portname 0,”PRI-0a” This identifies port 0 as being connected to port 0a on storage appliance PRI. Portname 8,“SHLF1-A-in” This identifies port 8 as being connected to A-IN on disk shelf 1. configshow Verify that the switch settings are correct. CABLE PRIMARY NODE Because of the architecture of MetroCluster, it is important to install cabling exactly as described. Although a Fibre Channel switch is quite flexible, Data ONTAP has specific requirements. Most installation problems are caused by cabling problems or misconfiguration of switches. In the cabling diagrams that follow, each cable is labeled with reference to the included chart. The diagram shows a FAS 3000, but other platform ports are included in the chart. If the installation includes more than two shelves at each site, refer to Section 4.4, "Adding More Shelves." MetroCluster Design & Implementation Guide – version 1.0 18 18 Slide 19: PRIMARY CONTROLLER TO SWITCHES 0 Local Node 6 F C / VI 5 2 3 1 Pool 0 Pool 1 4 SW1 7 Long Haul FC Pool 0 Pool 1 Pool SW2 8 Local Node Connection Chart Conn # 1 2 3 4 5 6 7 8 Type FC HBA/Port FC HBA/Port FC HBA/Port FC HBA/Port Cluster Cluster ISL ISL 9xx 5a 5b 8a 8b 6a 6b 3xxx 0a 0b 0c 0d 2a 2b 6xxx* 0a 0b 0e 0f 1a 1b SW#/ Port 1/0 2/0 1/1 2/1 1/12 2/12 1/13 2/13 * Refer to MetroCluster Compatibility Matrix for specific models Figure 4- 1) Primary node cabling – appliance to switches. MetroCluster Design & Implementation Guide – version 1.0 19 19 Slide 20: PRIMARY DISK SHELVES TO SWITCHES Local Node 2 Pool 0 Pool 1 SW1 1 4 Pool 0 Pool 1 SW2 3 OUT Pool A IN Conn # 1 2 3 4 5 6 7 8 9 10 Local Node Disk Shelf Connection Chart Shelf Type Desc SW#/Port ID P0-Loop1A 2/8 Local Loop 1 P0-Loop1B 1/8 P1-Loop1A 2/4 Remote Mirror 1 P1-Loop1B 1/4 P0-Loop2A 2/9 Local Loop 2 P0-Loop2B 1/9 P0-Loop3A 2/10 Local Loop 3 P0-Loop3b 1/10 P0-Loop4A 2/11 Local Loop 4 P0-Loop4B 1/11 IN B OUT A B Pool 0 OUT IN Pool 0 IN OUT OUT A B IN IN OUT OUT IN Pool 1 A IN B OUT Pool 1 Figure 4-2) Primary node cabling – disk shelf to switch. VERIFY CONNECTIONS Confirm that the disks are visible and have dual paths by entering the following command on the console: storage show disk -p The output shows the disks connected to the switches, the port to which they are connected, and the disk and module to which they belong, as shown in the following example: ha16> storage show disk -p PRIMARY PORT SECONDARY PORT SHELF BAY ------- ---- --------- -------- -switch3:0.40 A switch4:0.40 B 5 0 switch4:0.41 B switch3:0.41 A 5 1 . . . switch3:12.52 B switch4:12.52 A 6 4 If redundant paths are not shown for each disk, recheck the cabling. Note: When doing recabling or expansions, it is important to triple-check that the A-loop and B-loop of a shelf run to the very same port number on the two local switches. This is especially important when, for example, a 16-port 3850 switch gets replaced by a 200E. Older versions of Data ONTAP immediately panic and halt when the links to a shelf are not connected to the same port number. This affects both nodes of a fabric MetroCluster simultaneously, leading to a 100% outage for both nodes. SET UP MIRRORS To ensure highly available access to data, all data on each node must be mirrored to the other node using SyncMirror. Although it is possible to have data on one node that is not mirrored on the other, it is MetroCluster Design & Implementation Guide – version 1.0 20 20 Slide 21: not recommended. Keep in mind that mirrors can exist only between like drive types (FC to FC or ATA to ATA in the case of Stretch MetroCluster). With the SyncMirror license installed, disks are divided into pools. When a mirror is created, Data ONTAP pulls disks from Pool0 for the primary data and from Pool1 for the mirror. The selection of which disks to use for the mirror can be left up to Data ONTAP or chosen specifically. For detailed information, see he Data ONTAP Online Backup and Recovery Guide. It is important to verify the correct number of disks in each pool before creating the mirrored aggregate or traditional volume. Any of these commands can be used to verify the number of drives in each pool: Sysconfig -r -gives the broadest information Aggr status -r Vol status -r Once the pools are verified, mirrors can be created by one of the following: { aggr \| vol } create { aggrname \| volname} -m ndisks[@disk-size] For example, the command aggr create aggrA -m 6 creates a mirrored aggregate called aggrA with 6 drives (3 for plex 0, 3 for plex 1). Or an existing aggregate can be mirrored using the following: { aggr \| vol } mirror { aggrname \| volname } In both of the above cases, Data ONTAP is allowed to choose the specific drives to be used. Optionally, the user can choose the drives. Further information can be found in Data ONTAP Online Backup and Recovery. SOFTWARE DISK OWNERSHIP Software disk ownership is the method used for the FAS 3040/3070 and FAS 6000 Stretch MetroCluster to assign disk ownership. It will be supported on the fabric MetroCluster for these models when that configuration is released. It allows greater flexibility but requires greater thought in assigning disks. Two important items to remember are: • • Always assign all disks on the same loop to the same system and pool. Always assign all loops connected to the same adapter to the same pool. Use the disk show -n command to view all disks that do not have assigned owners. Use the following command to assign the disks that are labeled Not Owned to one of the system controllers. Note: If you are assigning unowned disks to a nonlocal storage system, you must identify the storage system by using either the -o ownername or the -s sysid parameter or both. disk assign {disk_name\|all\|-n count\|auto} [-p pool] [-o ownername] [-s sysid] [-c block\|zoned] [-f] disk_name[JW1] specifies the disks that you want to assign to the system. all specifies that all the unowned disks are assigned to the system. MetroCluster Design & Implementation Guide – version 1.0 21 21 Slide 22: -n count specifies the number of unassigned disks to be assigned to the system, as specified by count. auto causes disk auto-assignment to be performed. -p pool specifies which SyncMirror pool the disks are assigned to. The value of pool is either 0 or 1. Note: Unassigned disks are associated with a pool. To assign them to a different pool, use the -f option. However, moving individual disks between pools could result in the loss of redundancy and cause disk auto-assignment to be disabled for that loop. For this reason, you should move all disks on that loop to the other pool if possible. -o ownername specifies the system that the disks are assigned to. -s sysid specifies the system that the disks are assigned to. -c specifies the checksum type (either block or zoned) for a LUN in V-Series systems. -f must be specified if a system already owns the disk or if you want to assign a disk to a different pool. Enter man disk or refer to the Data ONTAP installation and administration guide for details. 4.2. Remote Site VERIFY LICENSES Follow the same procedures as performed at the primary site. CONFIGURE FIBRE CHANNEL SWITCHES Follow the same procedures as performed at the primary site. MetroCluster Design & Implementation Guide – version 1.0 22 22 Slide 23: CABLE REMOTE NODE REMOTE CONTROLLER TO SWITCHES Remote Node 5 6 F C / VI 2 4 Pool 0 Pool 1 3 1 SW3 7 Long Haul FC Pool 0 Pool 1 Pool 1 SW4 8 Remote Node Connection Chart Conn Type 9xx 3xxx # 6xxx 1 FC HBA/Port 5a 0a 0a 2 5b 0b 0b FC HBA/Port 3 8a 0c 0e FC HBA/Port FC HBA/Port 4 8b 0d 0f 5 Cluster 6a 2a 1a 6 Cluster 6b 2b 1b 7 ISL 8 ISL Figure 4-3) Remote node – appliance to switch. SW#/ Port 3/8 4/8 3/9 4/9 3/4 4/4 3/5 4/5 MetroCluster Design & Implementation Guide – version 1.0 23 23 Slide 24: REMOTE DISK SHELVES TO SWITCHES Remote Node 2 Pool 0 Pool 1 SW3 4 1 Pool 0 Pool 1 SW4 3 OUT Pool A B IN Conn # 1 2 3 4 5 6 7 8 9 10 Remote Node Disk Shelf Connection Chart Shelf Type Desc SW#/Port ID P0- Loop1A 4/0 Local Loop 1 P0- Loop1B 3/0 P1- Loop1A 4/12 Remote Mirror 1 P1- Loop1B 3/12 P0- Loop2A 4/1 Local Loop 2 P0- Loop2B 3/1 P0- Loop3A 4/2 Local Loop 3 P0- Loop3b 3/2 P0- Loop4B 4/3 Local Loop 4 P0- Loop4B 3/3 IN OUT Pool 0 OUT A B IN Pool 0 IN OUT OUT A IN IN OUT B OUT A B IN Pool 1 Pool 1 IN OUT Figure 4-4) Remote node cabling – disk shelf to switch. VERIFY CONNECTIONS Follow the same procedures as performed at the primary site. SET UP MIRRORS Follow the same procedures as performed at the primary site. SOFTWARE DISK OWNERSHIP Follow the same procedures as performed at the primary site. 4.3. Testing VERIFY THAT EACH SITE CAN SEE THE PROPER DISKS TEST FAILOVER • • Primary to Remote Remote to Primary TEST GIVEBACK • • Primary to Remote Remote to Primary MetroCluster Design & Implementation Guide – version 1.0 24 24 Slide 25: 4.4. Adding More Shelves Additional disk shelves can be added to a MetroCluster configuration according to the procedures defined in the Data ONTAP Active-Active Configuration Guide. The following guidelines apply: • • • ATA shelves are not supported on a Fabric MetroCluster. Make sure that additional shelves are added at the partner for mirroring. When adding shelves to new ports, the very same port on the other switch must be used. Otherwise, both nodes may not be boot (with invalid disk configuration), or they may panic and halt at once, if this is performed online. Only like shelves can be mirrored; for example, FC to FC or ATA to ATA (Stretch MetroCluster only). Limit of two shelves per switch port on a Fabric MetroCluster. Total limit for a Fabric MetroCluster is 336 disks. For a Stretch MetroCluster, the limit is that of the specific platform. When adding shelves to a Fabric MetroCluster, be sure to configure the associated switch ports to which the new shelves are connected. Keep in mind that each loop must have shelves of the same speed (2 or 4Gbps) and have the same type of controller (all LRC or all ESH/ESH2/ESH4). ESH4 disk shelves have a shorter maximum distance at 4Gbps. • • • • • • 16-port Switch Local 0 1 2 3 8 9 10 11 0 1 2 Remote 3 8 9 10 11 -----------------------------------4 5 6 7 Pool 0 --------------------------------12 13 14 15 -----------------------------------4 5 6 7 Pool 0 --------------------------------12 13 14 15 ------------------------------------ Pool 1 --------------------------------- ------------------------------------ Pool 1 --------------------------------- Figure 4-5) 16-port switch port assignments. Controller FC Port Disk Shelf Unused Interswitch Link FC/VI • 8-Port Switch Local 0 1 2 3 8 9 10 11 0 1 2 3 Remote 8 9 10 11 ---- Pool 0 ---4 5 --- Pool 1 --6 7 12 13 14 15 ---- Pool 0 ---- ---- Pool 1 ---4 5 6 7 12 13 14 15 Figure 4-6) 8-port switch port assignments. As shown in Figure 4.7a and b, additional shelves may can added with a maximum of two shelves per loop (Fabric MetroCluster). MetroCluster Design & Implementation Guide – version 1.0 25 25 Slide 26: Adding Storage to Primary Primary Switches 1,2 Pool 1 Pool 01 0 Secondary Pool 1 Pool 01 0 Switches 3,4 1st Loop Mirror 1st Loop 2 Shelves 2nd Loop 2 Shelves 3rd Loop 2 Shelves For Mirror 2nd Loop Mirror 3rd Loop Mirror Figure 4-7a) Adding disk shelves to primary node. Switches 1,2 1st Loop Mirror 2nd Loop Mirror 3rd Loop Mirror Primary Pool 1 Pool 01 0 Adding Storage to Secondary Secondary Pool 1 Pool 01 0 Switches 3,4 1st Loop 2 Shelves 2nd Loop 2 Shelves 3rd Loop 2 Shelves For Mirror NOTES 1. Maximum number of disks equals lesser of platform maximum or 336. 2. At each location , each shelf must be connected to the same port on each of the two switches . 3. See Figure 3-1 for physical shelf placement 4th Loop 2 Shelves Figure 4-7b) Adding disk shelves to remote node. 4.5. Best Practices and Recommendations The following recommendations can enable a timely and successful installation. CABLING, CABLING, CABLING Many people with SAN experience and knowledge of Fibre Channel switches assume that they can take advantage of the inherent flexibility that these switches typically provide. In the case of a Fabric MetroCluster configuration using hardware disk ownership, Data ONTAP expects specific devices on specific ports. Therefore it is important to follow the cabling diagram exactly. Full cabling diagrams are included in Appendixes E and F. CONFIGURATION OF STORAGE CONTROLLER FC PORTS Depending on any prior use of the NetApp storage controllers, it may happen that the on-board Fibre Channel ports are not configured correctly. This affects the controller’s ability to see all of the necessary disk shelves. To verify the configuration: Verify that the on-board ports are configured correctly: fcadmin config MetroCluster Design & Implementation Guide – version 1.0 26 26 Slide 27: Primary Adapter Type State Status --------------------------------------------------0a initiator UNDEFINED online 0b initiator UNDEFINED online 0c initiator UNDEFINED online 0d initiator CONFIGURED online They should all show up as initiator. For any that do not, perform the following. Ensure that the Fibre Channel ports are offline. fcadmin config -d <adapter> Note: Adapter refers to the port ID (for example, fcadmin config -d 0c took port 0c offline). Also, more than one port can be specified. Set the onboard port to target or initiator mode: fcadmin config -t initiator <adapter> Reboot the controller and check again: fcadmin config This verifies that the onboard Fibre Channel ports are online. CONFIGURATION OF SWITCHES • If the Brocade 200E switch is being used in the 8-port configuration, make sure that no SFPs are plugged into ports 8-15. Even though those ports may not be licensed or enabled, if Data ONTAP sees SFPs in those ports it treats it as a 16-port switch and expects to see devices connected accordingly. MetroCluster Design & Implementation Guide – version 1.0 27 27 Slide 28: 5. 5.1. Site Failover and Recovery Site Failover There are several situations that could necessitate a site takeover. They include: • • Complete environmental failure (air conditioning, power, etc.) Geographic disaster (earthquake, fire, flood, etc.) Upon determining that one of the sites has failed, the administrator must execute a specific command on the surviving node to initiate a site takeover. The command is: cf forcetakeover -d The takeover is not automatic because there may be cases in an active-active configuration where the network between sites is down and each site is still fully functional. In this case a forced takeover might not be desirable. It is important to remember that this is the case only when a complete site is lost. In the case of a failed controller at one of the sites, a normal cluster failover occurs. Due to the operation of SyncMirror, there is also added protection from multidrive or complete shelf failure. 5.2. Split Brain Scenario The cf forcetakeover command previously described allows the surviving site to take over the failed site’s responsibilities without a quorum of disks available at the failed site (normally required). Once the problem at the failed site is resolved, the administrator must follow certain procedures, including restricting booting of the previously failed node. If access is not restricted, a split brain scenario may occur. This is the result of the controller at the failed site coming back up not knowing that there is a takeover situation. It begins servicing data requests while the remote site also continues to serve requests. The result is the possibility of data corruption. You can restrict access to the previously failed site controller in the following ways: • • Turn off power to the previously failed node (disk shelves should be left on). Use manual fencing. 5.3. Recovery Process Although a complete site failover can be performed with a single command, there are cases where another step or two may be necessary before data is accessible at the surviving site. NFS Volumes You must remount the volumes of the failed node because the volumes are accessed through the surviving node. For more information about mounting volumes, see the File Access and Protocols Management Guide. iSCSI and Fibre Channel LUNS After a forced takeover, all LUNs that were being served from the failed site are served by the surviving site. However, each of these LUNs must be brought online. For example: lun online /vol/vol1/lun0 /vol/vol1/lun1 The reason they are offline is to avoid any LUN ID conflict. For example, suppose that two LUNs with ID of 0 mapped to the same igroup, but one of these LUNs was offline before the disaster. If the LUN that MetroCluster Design & Implementation Guide – version 1.0 28 28 Slide 29: was previously offline came online first, the second LUN would not be accessible because two LUNs with the same ID mapped to the same host cannot be brought online. Both of the steps described above could be automated with some clever scripting. The main challenge for the script is to understand the state of LUNs before the site failure so that the script brings online only LUNs that were online prior to the site failure. 5.4. Giveback Process After all problems causing the site failure have been resolved and you have ensured that the controller at the failed site is offline, it is time to prepare for the giveback so the sites can return to their normal operation. The following procedure can be used to resynchronize the mirrors and perform the giveback. COMMAND aggr status -r RESULT Validate that you can access the remote storage. If remote shelves don’t show up, power them and the switches on and check connectivity. After the node at the disaster site boots, it displays the following message: Turn on power to the node at the disaster site. Waiting for Giveback... partner Determine which aggregates are at the surviving site and which aggregates are at the disaster site by entering the following command: Go into partner mode on the remote node. Aggregates at the disaster site show plexes that are in a failed state with an out-of-date status. Aggregates at the surviving site show plexes as online. aggr status If aggregates at the disaster site are online, take them offline by entering the following command for each online aggregate: disaster_aggr is the name of the aggregate at the disaster site. Note: An error message appears if the aggregate is already offline. aggr offline disaster_aggr Re-create the mirrored aggregates by entering the following command for each aggregate that was split: aggr_name is the aggregate on the surviving site’s node. disaster_aggr is the aggregate on the disaster site’s node. The aggr_name aggregate rejoins the disaster_aggr aggregate to reestablish the MetroCluster configuration. Caution: Make sure that resynchronization is complete on each aggregate before attempting the following step. Return to the command prompt of the remote node. The node at the disaster site reboots. aggr mirror aggr_name -v disaster_aggr partner Enter the following command at the partner node: cf giveback MetroCluster Design & Implementation Guide – version 1.0 29 29 Slide 30: 6. 6.1. Relevant Documentation MetroCluster Documentation The following document can be found in the technical library: TR 3517: MetroCluster Upgrade Planning guide The following documents can be found on the NOW site: Active-Active Configuration Guide Data Protection Online Backup and Recovery Guide (Chapter 8, SyncMirror) MetroCluster Compatibility Matrix 6.2. Brocade Switch Documentation http://now.netapp.com/NOW/knowledge/docs/brocade/relbroc30_40/ MetroCluster Design & Implementation Guide – version 1.0 30 30 Slide 31: 7. 7.1. Appendixes Appendix A: Switch Ports Switch 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Switch 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Primary Node F800 F9xx F3xxx F6xxx 5a 5a 0a 0a 8a 8a 0c 0e Unusable Unusable Unusable Unusable Unusable Unusable Unusable Unusable P1-Loop1B P1-Loop1B P1-Loop1B P1-Loop1B P1-Loop2B P1-Loop2B P1-Loop2B P1-Loop2B P1-Loop3B P1-Loop3B P1-Loop3B P1-Loop3B P1-Loop4B P1-Loop4B P1-Loop4B P1-Loop4B P0-Loop1B P0-Loop1B P0-Loop1B P0-Loop1B P0-Loop2B P0-Loop2B P0-Loop2B P0-Loop2B P0-Loop3B P0-Loop3B P0-Loop3B P0-Loop3B P0-Loop4B P0-Loop4B P0-Loop4B P0-Loop4B FCVI 10a FCVI 6a FCVI 2a FCVI 1a ISL ISL ISL ISL Unusable Unusable Unusable Unusable Unusable Unusable Unusable Unusable F800 5b 8b Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A P0-Loop1A P0-Loop2A P0-Loop3A P0-Loop4A FCVI 10b ISL Unusable Unusable F9xx 5b 8b Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A P0-Loop1A P0-Loop2A P0-Loop3A P0-Loop4A FCVI 6b ISL Unusable Unusable F3xxx 0b 0d Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A P0-Loop1A P0-Loop2A P0-Loop3A P0-Loop4A FCVI 2b ISL Unusable Unusable F6xxx 0a 0e Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A P0-Loop1A P0-Loop2A P0-Loop3A P0-Loop4A FCVI 1b ISL Unusable Unusable Pool0/ Plex0 Pool1/ Plex1 Pool0 /Plex0 Pool1/ Plex1 Pool0/ Plex0 Pool1/ Plex1 Pool0/ Plex0 Pool1/ Plex1 Switch 3 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Switch 4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Remote Node F800 P0-Loop1B P0-Loop2B P0-Loop3B P0-Loop4B FCVI 10a ISL Unusable Unusable 5a 8a Unusable Unusable P1-Loop1B P1-Loop2B P1-Loop3B P1-Loop4B F800 P0-Loop1A P0-Loop2a P0-Loop3a P0-Loop4a FCVI 10b ISL Unusable Unusable 5b 8b Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A F9xx P0-Loop1B P0-Loop2B P0-Loop3B P0-Loop4B FCVI 6a ISL Unusable Unusable 5a 8a Unusable Unusable P1-Loop1B P1-Loop2B P1-Loop3B P1-Loop4B F9xx P0-Loop1A P0-Loop2a P0-Loop3a P0-Loop4a FCVI 2b ISL Unusable Unusable 5b 8b Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A F3xxx P0-Loop1B P0-Loop2B P0-Loop3B P0-Loop4B FCVI 2a ISL Unusable Unusable 0a 0c Unusable Unusable P1-Loop1B P1-Loop2B P1-Loop3B P1-Loop4B F3xxx P0-Loop1A P0-Loop2a P0-Loop3a P0-Loop4a FCVI 2b ISL Unusable Unusable 0b 0d Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A F6xxx P0-Loop1B P0-Loop2B P0-Loop3B P0-Loop4B FCVI 1a ISL Unusable Unusable 0a 0e Unusable Unusable P1-Loop1B P1-Loop2B P1-Loop3B P1-Loop4B F6xxx P0-Loop1A P0-Loop2a P0-Loop3a P0-Loop4a FCVI 1b ISL Unusable Unusable 0a 0e Unusable Unusable P1-Loop1A P1-Loop2A P1-Loop3A P1-Loop4A MetroCluster Design & Implementation Guide – version 1.0 31 31 Slide 32: 7.2. Appendix B: Switch Software Upgrade Procedure In order to ensure an optimum MetroCluster environment, it may occasionally be necessary to upgrade the firmware on the Fibre Channel switches. To determine the proper revision level, consult the MetroCluster Compatibility Matrix on the NOW site. The instructions in this section can also be found at http://now.netapp.com/NOW/download/software/sanswitch/fcp/Brocade/mc_ontap641_fabric_200_down load.shtml. This page provides access to the software installation package for Brocade 5.1.0 firmware. Downloading the Brocade 5.1.0 Firmware To download the Brocade 5.1.0 firmware to a workstation and then install it on a Brocade 200E switch, complete the following steps: 1. 2. 3. 4. 5. Ensure that the workstation recognizes the switch on the network (using the ping utility, for example). From the browser on the workstation, right-click this installation package (240.5 MB) link. Specify the directory to which you want to download the firmware installation package. After downloading is complete, access the switch from the workstation, using a Telnet session. Install the firmware on the switch by issuing the firmware download command. For the syntax and a description of the firmware download command, see the Brocade Fabric OS Reference Guide. Reboot the switch to activate the new firmware. 6. MetroCluster Design & Implementation Guide – version 1.0 32 32 Slide 33: 7.3. Appendix C: Fabric MetroCluster Worksheet The fabric MetroCluster worksheet is a planning aid where you can record specific cabling information about your fabric MetroCluster. You use this information during configuration procedures. Step 1 2 3 4 Hostname IP Address PRIMARY NODE (Primary) Cluster, Cluster_Remote, SyncMirror Licensed? Fibre Channel Switches 1 2 Switch License Keys? Full Fabric Ports on Demand (2) if 16-port switch Extended Distance (if >10km) Name IP Address Domain ID 5 6 Fibre Channel HBA 1 Port A Port B Slot # ____ Switch 1 Port # Switch 2 Port # 7 Fibre Channel HBA 2 Port A Port B Slot # ___ Switch 1 Port # Switch 2 Port # 8 Cluster Adapter Port # A B Switch 1 Port # Switch 2 Port # 9 10 Inter-Switch Link Disk Shelves Pool0 (Primary) A IN Pool0 (Primary) B IN Pool1 (Remote) A IN Pool1 (Remote) B IN Pool0 (Primary) A IN Pool0 (Primary) B IN Pool1 (Remote) A IN Pool1 (Remote) B IN Shelf ID Switch 1 Port # Switch 2 Port # MetroCluster Design & Implementation Guide – version 1.0 33 33 Slide 34: Step 1 2 3 4 REMOTE NODE (REMOTE) Hostname IP Address Cluster, Cluster_Remote, SyncMirror Licensed? Fibre Channel Switches 3 4 Switch License Keys? Full Fabric Ports on Demand (2) if 16-port switch Extended Distance (if >10km) Name IP Address Domain ID 5 6 Fibre Channel HBA 1 Port A Port B Slot # ____ Switch 3 Port # Switch 4 Port # 7 Fibre Channel HBA 2 Port A Port B Slot # ___ Switch 3 Port # Switch 4 Port # 8 Cluster Adapter Port # A B Switch 3 Port # Switch 4 Port # 9 10 Inter-Switch Link Disk Shelves Pool0 (Primary) A IN Pool0 (Primary) B IN Pool1 (Remote) A IN Pool1 (Remote) B IN Pool0 (Primary) A IN Pool0 (Primary) B IN Pool1 (Remote) A IN Pool1 (Remote) B IN Shelf ID Switch 3 Port # Switch 4 Port # MetroCluster Design & Implementation Guide – version 1.0 34 34 Slide 35: 7.4. Appendix D: Fabric MetroCluster (16-Port Switches) Local Node 5 1 F C / V I 2 3 4 6 Pool 0 Pool 1 SW1 7 Long Haul FC 9 Pool 0 Pool 1 Pool 1 0 OUT 10 SW2 8 A IN IN BOUT Pool 0 Local Node Connection Chart * Pool 0 Conn # 1 2 3 4 5 6 7 8 9 10 11 12 Type FC Disk FC Disk FC Disk FC Disk Cluster Cluster ISL ISL Local shelf A Local shelf B Remote shelf A Remote shelf B 9xx 5a 5b 8a 8b 6a 6b 3xxx 0a 0b 0c 0d 2a 2b 6xxx 0a 0b 0e 0f 1a 1b SW#/Po rt 1/0 2/0 1/1 2/1 1/12 2/12 1/13 2/13 2/8 1/8 2/4 1/4 11 12 OUT A IN IN B OUT OUT A B IN Pool 1 IN OUT OUT A B IN Pool 1 IN OUT Slide 36: Remote Node 5 6 1 2 Pool 0 Pool 1 Pool F C / V I 3 4 SW3 Long Haul FC 7 12 11 8 Pool 0 Pool 1 SW4 OUT A B IN IN OUT Pool 0 9 10 Remote Node Connection Chart* Conn # 1 2 3 4 5 6 7 8 9 10 11 12 Type FC HBA FC Disk FC Disk FC Disk Cluster Cluster ISL ISL Local Shelf A Local shelf B Remote Shelf A Remote Shelf B 9xx 5a 5b 8a 8b 6a 6b 3xx x 0a 0b 0c 0d 2a 2b 6xx x 0a 0b 0e 0f 1a 1b SW#/P ort 3/8 4/8 3/9 4/9 3/4 4/4 3/5 4/5 4/0 3.0 4/12 3/12 OUT A IN IN B Pool 0 OUT OUT A B IN Pool 1 IN OUT OUT A B IN Pool 1 IN OUT MetroCluster Design & Implementation Guide – version 1.0 36 36 Slide 37: 7.5. Appendix E: Fabric MetroCluster (8-Port Switches) Local Node F C / V I 6 34 Pool 0 Pool 1 1 2 5 SW1 7 Long Haul FC Pool 0 Pool 1 SW2 8 9 OUT A B IN Pool 0 11 10 IN OUT OUT A IN IN B Pool 0 12 OUT Conn# 1 2 3 4 5 6 7 8 9 10 11 12 Local Node Connection Chart* Type 9xx 3xxx 6xxx FC HBA 5a 0a 0a FC HBA 5b 0b 0b FC HBA 8a 0c 0e FC HBA 8b 0d 0f Cluster 6a 2a 1a Cluster 6b 2b 1b ISL ISL Local Shelf A Local Shelf B Remote Shelf A Remote Shelf B SW#/Port 1/0 2/0 1/1 2/1 1/6 2/6 1/7 2/7 2/4 1/4 2/2 1/2 OUT A B IN Pool 1 IN OUT OUT A B IN Pool 1 IN OUT © 2007 Network Appliance, Inc. All rights reserved. Specifications subject to change without notice. NetApp, the Network Appliance logo, Data ONTAP, and SyncMirror are registered trademarks and Network Appliance, NOW, and RAID-DP are trademarks of Network Appliance, Inc. in the U.S. and other countries. Windows is a trademark of Microsoft Corporation. UNIX is a registered trademark of The Open Group. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. MetroCluster Design & Implementation Guide – version 1.0 37 37