3 Managing Networks in a TruCluster Software Configuration

Member systems in a Production Server cluster employ networks in the following two ways:

The available server environment (ASE) daemons on cluster members communicate with each other using a single network associated with the MEMORY CHANNEL interconnect. This is how the ASE infrastructure uses the network.

Clients access ASE services over networks. This is how ASE services use networks.

The network used by the cluster infrastructure can be made more reliable by configuring redundant MEMORY CHANNEL connections between member systems, as described in the TruCluster Software Products Hardware Configuration manual. If one MEMORY CHANNEL connection fails, the daemons will communicate over the other MEMORY CHANNEL connection, which maintains cluster operation. The MEMORY CHANNEL network still appears to the ASE infrastructure as a single network.

Only when a member system cannot access other member systems over either MEMORY CHANNEL connection can a full network partition occur. If a full network partition occurs, the services continue to run on the member system and can be automatically failed over if the system crashes, but you cannot use the asemgr utility to change the ASE or to manually relocate services until the full network partition has been resolved.

Member systems in an Available Server configuration also use networks in two ways:

The ASE daemons on ASE members communicate with each other using a single network interconnect (Ethernet, FDDI, or ATM) to which all members are attached. An Available Server configuration does not use the MEMORY CHANNEL interconnect.

As in a Production Server cluster, clients access ASE services over networks. This is how ASE services use networks.

Using multiple networks in an Available Server configuration has the following advantages:

It allows you to increase the availability of applications and data. When you add a member system to an ASE, the asemgr utility prompts you for network interfaces for the member system. Configuring multiple network paths between member systems in an Available Server configuration reduces the chance that a member system will be erroneously considered unavailable.
Networks and shared SCSI buses are used to query member systems and determine their viability. If you configure multiple network connections between member systems, instead of querying only over the network associated with the member system name, TruCluster Available Server Software queries over backup networks. Use the asemgr utility to specify primary and backup networks.

If a network or network interface fails, member systems can still communicate over another network path, and ASE operation is not impaired. The ASE daemons on the member systems communicate using Transmission Control Program (TCP) connections over the network interface associated with the member system name. If you configure multiple network paths between member systems, and the path for the network interface associated with a member system name fails, the daemons will communicate over a backup network, which maintains ASE operation.
Only when a member system cannot access other member systems over any of its configured network interfaces does a full network partition occur. For example, a full network partition occurs if only one network is used in an ASE and that path fails, or if more than one network is used and all the paths fail. If a full network partition occurs, the services continue to run on the member system and can be automatically failed over if the system crashes, but you cannot use the asemgr utility to change the ASE or to manually relocate services.

In either a Production Server cluster or an Available Server configuration, client access to ASE services over networks can be made more reliable by monitoring specific network interfaces and taking specific actions (such as relocating services) when a particular interface fails. Monitor an interface if you are concerned with client access to ASE services on a particular interface. Monitoring an interface allows you to customize ASE operation when a network interface fails. See Section 3.3.5 for a discussion of how to monitor network interfaces.

3.1 Network Requirements

The network requirements for Production Server clusters are as follows:

Production Server clusters support only the MEMORY CHANNEL interconnect as a primary intracluster network.

Available server environment (ASE) member systems must be on at least one Internet Protocol (IP) network subnet that is common to the ASE.

Network interface names for common networks must be included in the local /etc/hosts file on each member system. See Section 3.2 for information.

The network requirements for an Available Server configuration are as follows:

TruCluster Available Server Software supports only Ethernet, FDDI, and ATM network hardware.

Member systems must be on at least one common IP network subnet.

Your primary and backup networks should be set up before you set up the TruCluster Available Server Software hardware and software.

Primary and backup networks in an Available Server configuration must be subnets that are common to all member systems. Network interface names for common networks must be included in the local /etc/hosts and /etc/routes files on each member system. See Section 3.2 for more information.

If the primary network connected to the systems becomes saturated, TruCluster Available Server Software operation is impaired. If you receive messages indicating that you are out of mbufs, adjust the ubcmaxpercent and ubcminpercent parameters in the configuration file. See the DIGITAL UNIX System Configuration and Tuning manual for more information.

3.2 Defining Network Interfaces

When you add a member system to an available server environment (ASE) in either a Production Server cluster or an Available Server configuration, the asemgr utility prompts you for additional network interface names. In an Available Server configuration, before you add an interface, you must use the netsetup utility to define the network interface on the system. The Production Server installation script automatically defines the network interface for the MEMORY CHANNEL interconnect on the system, so there is no need to run the netsetup utility to define its interface.

The following example is part of an /etc/hosts file on a Production Server cluster and shows two member systems, gideonmc and totomc, and multiple network interfaces for the systems:

# Cluster member systems (MEMORY CHANNEL interconnect)
#
10.0.0.1	gideonmc.abc.def.com gideonmc
10.0.0.2	totomc.adc.def.com totomc
#
#
# FDDI ring #1 (Client network 1)
#
16.142.112.121	gideonfddi1.abc.def.com gideon1
16.142.112.122	totofddi1.abc.def.com toto1
#
# FDDI ring #2 (Client network 2)
#
 
16.142.96.121	gideonfddi2.abc.def.com gideon2
16.142.96.122	totofddi2.abc.def.com toto2

The following example is part of an /etc/hosts file on an Available Server configuration and shows two member systems, gideon and toto, and multiple network interfaces for the systems:

# ASE member systems
#
16.140.64.121	gideon.abc.def.com gideon
16.140.64.122	toto.adc.def.com toto
#
#
# FDDI ring #1
#
16.142.112.121	gideon1.abc.def.com gideon1
16.142.112.122	toto1.abc.def.com toto1
#
# FDDI ring #2
#
16.142.96.121	gideon2.abc.def.com gideon2
16.142.96.122	toto2.abc.def.com toto2

In an Available Server configuration, you must specify the interface names for the primary and backup networks in the local /etc/routes file on each member system. For each member system, you must define a host route to all other member systems. This definition is needed to fail over IP traffic between member systems when a network path fails.

For example, if your member systems are gideon1 and toto1, where the number in the name refers to the subnet, and each member system also has interface names gideon2 and toto2, then each member system's /etc/routes file must contain the following information:

-host gideon1 gideon1
-host gideon2 gideon2
-host toto1 toto1
-host toto2 toto2

See routes(4) for information on the file format.

3.3 Modifying the Network Configuration

To modify the network configuration, choose the "Modify the network configuration" item from the Managing the ASE menu.

The ASE Network Modify Menu allows you to do the following:

Display the current network configuration (Section 3.3.1)

Add and delete network interfaces (Section 3.3.2)

Specify the primary network (Available Server configurations only) (Section 3.3.3)

Specify backup networks (Available Server configurations only) (Section 3.3.3)

Specify networks to ignore (Available Server configurations only) (Section 3.3.4)

Specify network interfaces to monitor (Section 3.3.5)

The following sections describe how to display and modify the network configuration.

3.3.1 Displaying the Network Configuration

Choose the "Show the current configuration" item from the ASE Network Modify Menu to display the member systems, their interface names, whether monitoring is enabled, or, in an Available Server configuration, whether an interface is connected to a primary or a backup network.

The following example shows the network configuration of a Production Server cluster:

                ASE Network Configuration
 
    Member Name          Interface Name       Monitor
    ___________          ______________       _______
    totomc               totomc               No
    totomc               totofddi1            Yes
    totomc               totofddi2            No
 
    gideonmc             gideonmc             No
    gideonmc             gideonfddi1          Yes
    gideonmc             gideonfddi2          No

The following example shows the network configuration of an Available Server configuration:

                ASE Network Configuration
 
    Member Name          Interface Name       Member Net  Monitor
    ___________          ______________       __________  _______
    toto                 toto                 Primary     Yes
    toto                 toto1                Backup      No
    toto                 toto2                Backup      No
 
    gideon               gideon               Primary     Yes
    gideon               gideon1              Backup      No
    gideon               gideon2              Backup      No

3.3.2 Adding and Deleting Network Interfaces

Before you specify a network interface for a member system, the interface must be defined and configured on the system. (See Section 3.2 for more information.)

Choose the "Add network interfaces" item from the ASE Network Modify Menu to add a network interface. From the ASE Member Menu, choose the number of the member to which you want to add a network interface. For example, on a Production Server cluster:

                ASE Member Menu
 
Select a member to add an interface to:
 
    0)  gideonmc
    1)  totomc
 
    q)  Quit without making changes
 
Enter your choice: 1

Enter interface names for member 'totomc'  
    Interface name (return to exit): totofddi1

To delete network interfaces, choose the "Delete network interfaces" item from the ASE Network Modify Menu. For example, on an Available Server configuration:

                ASE Member Menu
 
Choose a member to delete an interface from:
 
    0)  gideon
    1)  toto
 
    q)  Quit without making changes
 
Enter your choice: 1
 
         Network Interfaces for Member 'toto'
 
Choose one or more network interfaces to delete:
 
     )  toto     16.142.112.121       Not an option
    1)  toto1    16.142.112.122
    2)  toto2    16.142.96.122
 
    q)  Quit to previous menu
 
Enter your choices (comma or space separated):1

In an Available Server configuration, note that the member network interface cannot be deleted, regardless of whether it is defined as a primary or backup network. The member network interface is defined during software installation and establishes a system's membership in an ASE and its member name. For that reason, the member network interface name does not appear in the list of interfaces eligible for deletion. Similarly, in a Production Server cluster, you cannot delete a member's MEMORY CHANNEL interface.

3.3.3 Specifying Primary and Backup Networks (AS)

In an Available Server configuration, the primary network is the network that is used most frequently to query other member systems. Backup networks are also used for queries, but at a slower rate. Interfaces for primary and backup networks must be common to all the member systems and included in each member system's local /etc/hosts and /etc/routes files. See Section 3.2 for more information.

Choose the "Specify the primary ASE member network" item from the ASE Network Modify Menu to select an interface for the primary network. For example:

             ASE Member Primary Network Menu
 
Choose one of the networks to be the ASE member primary network:
 
    0)  16.142.112.0     (toto1, gideon1)
    1)  16.142.96.0      (toto2, gideon2)
 
    q)  Quit to previous menu
 
Enter your choice: 0

Choose the "Specify a backup ASE member network" item from the ASE Network Modify Menu to select backup network interfaces for the ASE. For example:

                ASE Member Backup Network Menu
 
Choose one of the networks to be the ASE member backup network:
 
    0)  16.142.112.0     (toto1, gideon1)
    1)  16.142.96.0      (toto2, gideon2)
 
    q)  Quit to previous menu
 
Enter your choices (comma or space separated): 1 
 
        16.142.96.0      (toto2, gideon2)
 
Are the above choices correct (y|n)? [y]:y

3.3.4 Specifying a Network to Ignore (AS)

In an Available Server configuration, choose the "Specify an ASE member network to be ignored" item from the ASE Network Modify Menu to specify a network that you want to configure but you do not currently want the member system to use. For example:

                Ignore ASE Member Network Menu
 
Choose a network not to be used as an ASE member network:
 
 
    0)  16.142.112.0     (toto1, gideon1)
    1)  16.142.96.0      (toto2, gideon2)
 
    q)  Quit to previous menu
 
Enter your choices (comma or space separated): 0
 
        16.142.112.0     (toto1, gideon1)
 
Are the above choices correct (y|n)? [y]:n

3.3.5 Monitoring Network Interfaces

You can monitor any network interface or any member system and take specific actions (such as relocating services, or sending mail or a page to an administrator) when a particular interface fails. Monitor those interfaces that are critical to clients accessing services. TruCluster software allows you to monitor up to four interfaces per member system at the same time.

In a Production Server cluster, you can monitor a member's MEMORY CHANNEL interfaces or any network interface that allows client access to the cluster's services. Similarly, in an Available Server configuration, you can monitor a member's primary and backup network interfaces. In either a cluster or an Available Server configuration, you can monitor a network interface that is not on a subnet common to all member systems.

If a monitored network interface fails, the TruCluster software runs the error Alert script (see Section 12.1.5), which invokes the member's /var/ase/lib/ni_status_awk script. By default, if all monitored network interfaces on the member are down, the /var/ase/lib/ni_status_awk script stops all the services running on that member and starts them on another member.

However, you can customize the /var/ase/lib/ni_status_awk script on each member system to specify a different action to take. For example, you can edit the script so that services relocate to another member system if any network interface fails or if a particular interface fails. In addition, because the error Alert script is propagated on all the member systems, you can edit the error Alert script itself, so that the actions will be the same on all systems. Use the asemgr utility to edit the error Alert script.

Choose the "Specify network interfaces to be monitored" item from the ASE Network Modify Menu to monitor specific interfaces. For example, on an Available Server configuration:

                ASE Member Menu
 
Choose a member to modify:
 
    0)  gideon
    1)  toto
    q)  Quit without making changes
 
Enter your choice: 0
 
           Network Interfaces for Member 'toto'
 
Choose one or more network interfaces:
 
    0)  toto           16.140.64.122         (monitored)
    1)  toto1          16.140.112.122        (monitored)
    2)  toto2          16.140.96.122         (monitored)
 
    q)  Quit to previous menu
    n)  Do not monitor any interfaces
 
Enter your choices (comma or space separated): 1
 
        toto1	 16.140.112.122
 
Are the above choices correct (y|n)? [y]: y

3.4 Using Multiple Client Networks and ASE Services

In either a Production Server cluster or an Available Server configuration, member systems, Network File System (NFS) services, tape services, and disk services that have IP addresses use the networking subsystem. The following sections apply only if your member systems are connected to more than one client network. If subnets are used, the term network is used in the following sections to refer to a subnet.

3.4.1 How ASE Services Use Multiple Networks

You can connect the member systems in either a Production Server cluster or an Available Server configuration to several client networks. All the member systems must be able to access each network, so that clients react correctly when the TruCluster software relocates an NFS or tape service (or a disk service that has an IP address).

Between the networks, there should be a separate router system that is not a member system. Do not use a member system as a general-purpose IP router, because system performance will be unpredictable.

To enable clients to access an NFS or tape service (or disk service that has an IP address), the service name is assigned its own Internet address. The service name that you choose must be native to one of the networks. On that network, the Address Resolution Protocol (ARP) translates the Internet address associated with the service name to the hardware address of the member running the service. If the service is relocated, the ARP translates that Internet address to the Ethernet address of the new server. Therefore, the ARP broadcasts enable clients to recognize when a service has relocated to a different member system.

After they receive the new ARP address translation, clients on the network that is native to the service name will start to send data to the new member system that is running the service. Clients on a network that is not native to the service name forward their packets through the router system to the network that is native to the service name.

The router system processes the ARP broadcasts sent from the member systems. Clients that are not on the native network should know how to send data through a router to the service name address. Clients that are on the native network only need to know how to react to ARP broadcasts.

3.4.2 Getting Faster Access to Services

If a client is not on the network that is native to an NFS or tape service name (or a disk service name that has an IP address), the client must send packets through the router system to reach the service address. Network traffic must go through an extra hop to access the service because the packets are forced to pass through an extra system.

A TruCluster Available Server Software feature that enables you to bypass this step requires that clients on a network that is not native to the service name use the Routing Information Protocol (RIP) routing protocol and respond to host routes. This feature broadcasts host routes on networks that are not native to the service name. The technique that ARP uses to handle service relocations is still used on the network that is native to the service name.

Host routes direct the clients to the member system that is running the service, without requiring clients to send data through the router system. Using this method, member systems do a restricted form of routing. Only host routes associated with NFS, tape, or disk service names are advertised.

You must manually enable this feature. If this feature is not enabled, and you have multiple networks and a separate router system, clients on all networks will react correctly to service relocations. However, some network traffic will require an extra step to reach the service.

To enable this feature, perform the following tasks:

Run the netsetup script on all systems.

Choose the "Enable/Disable Network Daemons and Add Static Routes" menu items.

Enter yes at the prompt that asks if you want to be an IP router.

Choose the gated option and do not specify any flags.

Choose the "Exit" menu item.

Do not restart the network services.

Kill the routed daemon if it is running.

Enter the following command:
```
# rcmgr set ASEROUTING yes
```

The ASEROUTING configuration variable allows for host-based routes from a server that has multiple network interfaces. This can make for faster connections to a service by avoiding routers and making use of the multiple interfaces.

However, only clients that can can listen to dynamic routing updates sent with the RIP routing protocol will benefit from setting ASEROUTING=yes. Clients that simply specify a default router will not benefit.

Note

You can use the ASEROUTING configuration variable only with the old gated daemon (ogated). (ogated is the default selection in the netsetup script.) If you use the ASEROUTING configuration variable when the new gated daemon is running on ASE members, all service operations will fail and error messages are entered in the daemon.log file.

Setting ASEROUTING to yes results in modifications to the /etc/ogated.conf files on all ASE members. If you have modified the /etc/ogated.conf files on ASE members, these changes might interfere with ASEROUTING behavior. Therefore, if you customize the /etc/ogated.conf files on ASE members, do not use the ASEROUTING option.

If you created services before you enabled this feature, you must modify all the services; this will delete the services, add the services, and start the advertising of the host-based routes.