H3C CAS CVM

Stateful Failover System Reconfiguration Guide

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Software version: E0523 and later versions

Document version: 5W100-20250331

The information in this document is subject to change without notice.

Contents

About this document 1

Change the VIP of the system·· 1

Method 1 (applicable when the stateful failover service can be stopped) 1

Method 2 (applicable when the stateful failover service cannot be stopped) 1

Replace the quorum node for advanced quorum·· 2

Restrictions and guidelines· 2

Replace the quorum node collocated with a CVK node within the cluster, with its IP and host name unchanged  2

Replace the quorum node collocated with a CVK node within the cluster, with its IP or host name changed  3

Replace an external quorum node with a CVK node for advanced quorum within the cluster 3

Change the quorum IP addresses for ping quorum·· 3

Change the IP address of the primary, backup, or quorum node· 4

Restrictions and guidelines· 4

Procedure· 4

Replace the IPv4 addresses of the nodes in the stateful failover system with IPv6 addresses· 6

Expand the CVM platform from a single-node deployment to a two-node deployment 8

Restrictions and guidelines· 8

Check and back up data before making changes· 8

Procedure· 9

Edit the host name of a node in the stateful failover system·· 9

About this task· 9

Procedure· 9

Replace advanced quorum with ping quorum·· 10

Prerequisites· 10

Restrictions and guidelines· 10

Procedure· 10

 

 

 

 

 

 

 

 

 

 

 

About this document

This document is intended to help the support personnel maintain the CVM stateful failover system for business continuity. It describes the methods to edit the virtual IP (VIP) of the stateful failover system, replace the quorum node, and change the IP addresses and host names of the nodes in the system.

Change the VIP of the system

The method to edit the VIP of the system depends on whether the stateful failover service can be stopped during maintenance. If you may stop the stateful failover service, use method 1. Whichever method you use, log in to the nodes in the system by using their real IP addresses.

Method 1 (applicable when the stateful failover service can be stopped)

1.     Execute the service cmsd stop command on the backup, primary, and quorum nodes in turn to shut down the stateful failover service.

2.     Execute the ip addr show vswitch0 command on the primary node to verify that the original VIP is not in the address table.

3.     Ping the planned new VIP to verify that it is not in use. If the IP address cannot be pinged, it is not in use. You can use it as the new VIP.

4.     Edit the configuration file at /etc/cmsd/services/network.conf on the primary, backup, and quorum nodes to replace the original VIP with the new VIP.

5.     Execute the service cmsd start command on the primary, backup, and quorum nodes to start the stateful failover service.

6.     Execute the ip addr show vswitch0 command on the primary node to verify that the new VIP is in effect. Then, verify that you can access the CVM Web interface at the new VIP.

Method 2 (applicable when the stateful failover service cannot be stopped)

1.     Verify that the stateful failover service is operating correctly. Execute the crm status command on the primary node to verify that the nodes in the stateful failover system are operating correctly. Then, execute the drbd-overview command to verify that the DRBD split-brain issue is not present.

2.     Ping the planned new VIP to verify that it is not in use. If the IP address cannot be pinged, it is not in use. You can use it as the new VIP.

3.     Execute the ms_service_network.sh stop command on the primary node to stop the management network for the stateful failover system.

4.     Edit the configuration file at /etc/cmsd/services/network.conf on the primary, backup, and quorum nodes to replace the original VIP with the new VIP.

5.     Execute the ms_service_network.sh start command on the primary node to start the management network for the stateful failover system.

6.     Execute the service cmsd start command on the quorum node to restart the stateful failover service.

7.     Execute the ip addr show vswitch0 command on the primary node to verify that the new VIP is in effect. Then, verify that you can access the CVM Web interface at the new VIP.

 

NOTE: ·     As a best practice, use the service cmsd stop and service cmsd start commands instead of the ms_service_network.sh stop and ms_service_network.sh start commands if the stateful failover service can be stopped without impacting services. ·     If the deployment does not use advanced quorum, ignore the operations on the quorum node.

 

Replace the quorum node for advanced quorum

Restrictions and guidelines

As a best practice, deploy the quorum node for advanced quorum on a CVK node managed by CVM for passwordless access. If the quorum node is an external node, you must configure that node for passwordless access. For instructions on configuring a node for passwordless access, contact technical support.

Replace the quorum node collocated with a CVK node within the cluster, with its IP and host name unchanged

This task is applicable to a stateful failover system that uses a CVK node within the cluster as the quorum node. After you format and reinstall that CVK node with its IP address and host name unchanged, you can perform this task to reconfigure it as the quorum node.

1.     Re-add the CVK host to the cluster.

2.     Copy the /etc/cmsd directory from the primary node to the /etc/ directory on the new quorum node.

3.     Edit the configuration file at /etc/cmsd/cmsd.xml on the new quorum node. Replace all occurrences of the self string with the host string. In addition, replace the host string at the same line as the role="3" key-value pair with the self string.

4.     Execute the service cmsd start command on the new quorum node to start the stateful failover service.

5.     Configure the new quorum node to start the stateful failover service at each startup:

¡     On a Ubuntu edition node, execute the following commands:

update-rc.d cmsd defaults 90

echo "* *    * * *   root    /opt/bin/ms_cmsd_realtime_check.sh" >> /etc/crontab

service cron restart

¡     On an H3Linux edition node, execute the systemctl enable cmsd.service command.

Replace the quorum node collocated with a CVK node within the cluster, with its IP or host name changed

This task is applicable to a stateful failover system that uses a CVK node within the cluster as the quorum node. After you change the IP address or host name of that CVK node, you can perform this task to reconfigure it as the quorum node.

1.     Edit the configuration file at /etc/cmsd/cmsd.xml on both the primary and backup nodes. Identify the name and ip attributes at the same line as the role="3" key-value pair. Replace the values for the name and ip attributes with the new host name and IP address of the new quorum node, respectively.

2.     Copy the /etc/cmsd directory from the primary node to the /etc/ directory on the new quorum node.

3.     Edit the configuration file at /etc/cmsd/cmsd.xml on the new quorum node. Replace all occurrences of the self string with the host string. In addition, replace the host string at the same line as the role="3" key-value pair with the self string.

4.     Execute the service cmsd start command on the new quorum node to start the stateful failover service.

5.     Execute the service cmsd stop command on the backup node to stop the stateful failover service.

6.     Execute the service cmsd restart command on the primary node to restart the stateful failover service.

7.     Execute the service cmsd start command on the backup node to start the stateful failover service.

Replace an external quorum node with a CVK node for advanced quorum within the cluster

To replace the external quorum node in a stateful failover system with a CVK node in the cluster for advanced quorum, perform this task.

1.     Perform all steps in "Replace the quorum node collocated with a CVK node within the cluster, with its IP or host name changed."

2.     Configure the new quorum node to start the stateful failover service at each startup:

¡     On a Ubuntu edition node, execute the following commands:

update-rc.d cmsd defaults 90

echo "* *    * * *   root    /opt/bin/ms_cmsd_realtime_check.sh" >> /etc/crontab

service cron restart

¡     On an H3Linux edition node, execute the systemctl enable cmsd.service command.

Change the quorum IP addresses for ping quorum

To change one or two quorum IP addresses for ping quorum:

1.     Edit the configuration file at /etc/cmsd/cmsd.xml on both the primary and backup nodes. Identify the ip and ip_other attributes at the same line as the role="3" key-value pair. Set the ip and ip_other attributes to the new quorum IPs.

2.     Execute the service cmsd stop command on the backup node to stop the stateful failover service.

3.     Execute the service cmsd restart command on the primary node to restart the stateful failover service.

4.     Execute the service cmsd start command on the backup node to start the stateful failover service.

Change the IP address of the primary, backup, or quorum node

Restrictions and guidelines

·     If the primary or backup CVM node is collocated with a CVK node on the same host, you must migrate all VMs on the host to another host before you remove the host from the cluster. After you change the IP address of the CVM node and add its host back to the cluster, migrate the VMs back.

·     On a node in Euler, the configuration file name differs depending on whether the management network uses VLANs. Additionally, the node will have multiple configuration files if VM traffic is separated from host traffic. If you are unsure about the traffic separation status, check for the configuration files listed in the following procedure and edit them if they exist.

Procedure

1.     Execute the ms_resource_stop.sh command on the primary node to stop the stateful failover service on the primary, backup, and quorum nodes.

2.     Change the IP address of the primary, backup, or quorum node as needed.

a.     Edit the network configuration files on the primary, backup, and quorum nodes.

-     In Ubuntu, edit the file at /etc/network/interfaces to set the new IP address. The following is a sample configuration:

auto vswitch0

iface vswitch0 inet6 static

      address 172.16.60.31

      netmask 24

      gateway 172.16.60.1

-     In CentOS, edit the file at /etc/sysconfig/network-scripts/ifcfg-vswitch0 to set the new IP address. The following is a sample configuration:

IPADDR="172.16.60.31"

PREFIX="24"

GATEWAY="172.16.60.1"

-     In Euler, if the management network is not VLAN-based, edit the file at /etc/sysconfig/network-scripts/ifcfg-vswitch0 to set the new IP address. If the management network is VLAN-based, edit the file at /etc/sysconfig/network-scripts/ifcfg-vswitch0.$vlan to set the new IP address.

 

 

IMPORTANT: ·     If the configuration file for a node contains multiple IP addresses, change the management IP address and keep other IP addresses unchanged. ·     The $vlan variable represents a VLAN value. ·     When you edit the ifcfg-vswitch0.$vlan file, you only need to change the IP address, just as you do when you edit the ifcfg-vswitch0 file.

 

The following is an example:

IPV6INIT="yes"

MTU="1500"

ALTNAME="eth2"

GATEWAY="172.16.60.1"

IPADDR="172.16.60.31"

PREFIX="24"

ARPCHECK="no"

ARPUPDATE="yes"

IPADDR0="2.1.1.1"

PREFIX0="16"

ARPCHECK0="no"

If the Euler system separates VM traffic from host traffic, the node will have a configuration file at /etc/linux-engine-networks/vswitch/vswitch0. Check for this file. If the file exists, change the management IP address in it.

The following is an example:

name=vswitch0

accele_mode=normal

network_type=1

vlan=1

mtu=1500

ipv4=172.16.60.31/24

gateway4=172.16.60.1

ipv6=

gateway6=

uplinks=eth2

bond_mode=

bond_policy=

b.     Execute the ifdown vswitch0 command and then the ifup vswitch0 command on the primary, backup, and quorum nodes to restart their NICs.

3.     On each of the primary, backup, and quorum nodes, change their IP addresses in the stateful failover configuration file to the new IP addresses.

¡     Edit the configuration file at /etc/cmsd/cmsd.xml on each of the primary, backup, and quorum nodes to change their IP addresses.

¡     Change the VIP of the stateful failover system if it becomes inaccessible after the IP address of the primary, backup, or quorum node changes. To change the VIP, open the configuration file at /etc/cmsd/services/network.conf on each of the primary, backup, and quorum nodes and edit the VIP and its mask. The following is a sample configuration:

ip = 172.16.60.32

netmask = 64

¡     On the primary and backup nodes, edit the file at /etc/drbd.d/r0.res to change all IP addresses to the new IP addresses. If a file exists at /etc/drbd.d/r1.res, edit it as well. The r1.res file is the partition synchronization configuration file for template pools. It exists only if you have configured template pools. Skip this step on the quorum node. The following is a sample configuration:

address 172.16.60.32:20136

4.     With E0783 or a later version, access the primary node, and then edit the database configuration file at /etc/database/pg_hba.conf to update the IP addresses and masks for the primary and backup nodes. Set the mask length to 32 for IPv4 and to 128 for IPv6. The following is a sample configuration:

host    all     ssadmin       172.16.60.31/32        md5

host    all     ssadmin       172.16.60.32/32        md5

host    all     ssadmin_ro    172.16.60.31/32        md5

host    all     ssadmin_ro    172.16.60.32/32        md5

5.     Execute the service cmsd start command on the primary, backup, and quorum nodes in turn to start the stateful failover service on them.

6.     Execute the crm status command on any of these nodes to verify that the stateful failover service is operating correctly.

Replace the IPv4 addresses of the nodes in the stateful failover system with IPv6 addresses

1.     Execute the ms_resource_stop.sh command on the primary node to stop the stateful failover service on the primary, backup, and quorum nodes. Then, verify that the stateful failover service has been successfully stopped, as follows:

¡     Execute the pidof cmsd command on the primary and backup nodes. If the stateful failover service is disabled, you receive an empty result.

¡     Execute the ip addr show vswitch0 command on the primary node. If the stateful failover service is disabled, the returned results do not include a VIP.

¡     Execute the drbd-overview command on the primary and backup nodes. If the stateful failover service is disabled, the system displays information similar to the following message:

0:r0/0  Unconfigured….

2.     Configure new IPv6 addresses in the configuration file on each of the primary, backup, and quorum nodes:

a.     Edit the configuration file on each of the primary, backup, and quorum nodes.

-     In Ubuntu, edit the file at /etc/network/interfaces to add the new IPv6 address for the node. The following is a sample configuration:

iface vswitch0 inet6 static

      address 172:20:28::91

      netmask 98

      gateway 172:20:28::1

-     In CentOS, edit the file at /etc/sysconfig/network-scripts/ifcfg-vswitch0 to add the new IPv6 address.

The following is an example:

IPV6ADDR="172:20:28::91/98"

IPV6_DEFAULTGW="172:20:28::1"

-     In Euler, if the management network is not VLAN-based, edit the file at /etc/sysconfig/network-scripts/ifcfg-vswitch0.

If the management network is VLAN-based, edit the file at /etc/sysconfig/network-scripts/ifcfg-vswitch0.$vlan.

Check for the /etc/linux-engine-networks/vswitch/vswitch0 file. If the file exists, edit the file to add the new IPv6 address.

The following is an example:

IPV6ADDR="172:20:28::91/98"

IPV6_DEFAULTGW="172:20:28::1"

b.     Execute the ifdown vswitch0 command and then the ifup vswitch0 command on the primary and backup nodes to restart their NICs.

3.     Edit the stateful failover service configuration files on each of the primary, backup, and quorum nodes to change their IP addresses to the new IPV6 addresses.

¡     Change all IP addresses in the configuration file at /etc/cmsd/cmsd.xml. Take changing one IP address for example:  

ip="172:20:28::91"

¡     Edit the file at /etc/cmsd/services/network.conf to change the VIP and its mask. The following is a sample configuration:

ip = 172:20:28::90

netmask = 64

¡     Edit the file at /etc/drbd.d/r0.res to change all IP addresses to the new IP addresses. If a file exists at /etc/drbd.d/r1.res, edit it as well. Skip this step on the quorum node. The following is a sample configuration:

address ipv6 [172:20:28:91]:20136

4.     With E0783 or a later version, access the primary node, and then edit the database configuration file at /etc/database/pg_hba.conf to update the IP addresses and masks for the primary and backup nodes. Set the mask length to 32 for IPv4 and to 128 for IPv6. The following is a sample configuration:

host    all     ssadmin       172.16.60.31/32        md5

host    all     ssadmin       172.16.60.32/32        md5

host    all     ssadmin_ro    172.16.60.31/32        md5

host    all     ssadmin_ro    172.16.60.32/32        md5

5.     Execute the ms_resource_start.sh command on the primary node to start the stateful failover service on the primary, backup, and quorum nodes. Then, verify that the stateful failover service has been successfully enabled, as follows:

¡     Execute the pidof cmsd command on the primary and backup nodes. If the stateful failover service is enabled, you receive stateful failover service information.

¡     Execute the ip addr show vswitch0 command on the primary node. If the stateful failover service is enabled, the returned results include the IPv6 VIP.

¡     Execute the drbd-overview command on the primary and backup nodes. If the stateful failover service is enabled, the system displays information similar to the following message:

0:r0/0  Connected  Primary/Secondary  UpToDate/ UpToDate C r-----/var/lib/mysql-share ext4 19G 349M 18G 2%

¡     Verify that the CVM Web interface is accessible.

Expand the CVM platform from a single-node deployment to a two-node deployment

Restrictions and guidelines

·     This task requires disk formatting. Do not perform this task to expand the CVM platform from a single-node deployment to a two-node deployment unless requested by the customer.

·     This task will result in the loss of performance monitoring data for the cluster, hosts, and VMs in the original CVM platform. If performance monitoring data is critical, contact technical support for assessment before you perform this task.

·     After you redeploy the stateful failover system, CVM might display CVK exceptions when it tests the passwordless access to CVK nodes. To resolve this issue, reconnect to the CVK nodes.

·     If the original CVM platform is installed with a patch, install the same patch on the new CVM platform to ensure their software version consistency before you perform CVM backup and restoration.

·     If the original CVM platform is integrated with CloudOS, change the address of CVM on CloudOS to the VIP of the current CVM platform.

Check and back up data before making changes

Before you expand the CVM platform from a single-node deployment to a two-node deployment for stateful failover, perform the following tasks:

1.     If the CVM platform has VM templates stored locally, copy the template files to another host or shared storage. After you deploy the stateful failover system, copy them back.

2.     Contact technical support to unbind its licenses and re-register the licenses to obtain activation files for stateful failover deployment. The licenses on a single-node management platform are not applicable after you expand the platform to a stateful failover system.

3.     Check the host that deploys the single-node CVM platform for a collocated CVK node. If a CVK node exists, migrate its VMs to another host. Then, delete the shared file system and shared storage pools on the local host, and remove the CVK node from the CVM platform.

4.     Disable HA, DRS, and scheduled policies for all clusters in CVM, and disable calls from external applications, such as an external backup application.

5.     Before you make changes, make sure the service network for CVM is operating correctly and all VMs in it are operating correctly as well.

6.     Back up CVM and then copy the backup files to another location. Manually back up essential VM data and take screenshots of critical webpage information to aid in service restoration after the stateful failover system is deployed.

7.     Before you make changes, check the following items for anomalies. For more information about the check methods, contact technical support.

¡     Host state and vSwitch states—Verify that the hosts and vSwitches in the cluster are operating correctly.

¡     VM states—Verify that all the VMs on the host are operating correctly. Record the operating state of each VM. If a VM is operating incorrectly, identify and resolve the issues to ensure a successful setup of the stateful failover system. If you encounter any issues, contact technical support.

Procedure

IMPORTANT: To expand a CVM platform from a single-node deployment to a two-node deployment for stateful failover, you must reinstall a new CVM host as required. For more information about setting up a CVM host, see H3C CAS CVM Stateful Failover Service Configuration Guide.

 

1.     Install CAS software on a new host. Make sure this host run the same software version as the existing CVM host. Then, expand the CVM platform to a stateful failover system. For more information about the deployment procedure, see H3C CAS CVM Stateful Failover Service Configuration Guide.

2.     After the stateful failover system is deployed, verify that all CVK hosts in the current environment have connectivity. Then, restore CVM. Carefully read the failure recovery instructions on the page during CVM restoration.

3.     After CVM restores, verify that each host is in normal state and has connectivity. Then, enable HA.

4.     Import the VM templates and perform license tasks as required.

Edit the host name of a node in the stateful failover system

About this task

To change the host name of the primary, backup, or quorum node (if any) in a stateful failover system, perform this task.

Procedure

1.     As a best practice, back up CVM before you edit the host name of a node in a stateful failover system.

 

IMPORTANT: If the target host also acts as a CVK node, migrate all VMs from the CVK host to another host. Then, delete the shared file system and shared storage pools on the target host, and remove the target host from CVM. After you change the host name of the target host, add it back to the cluster and then migrate the VMs back.

 

2.     Execute the service cmsd stop command on the backup, primary, and quorum nodes in turn to stop the stateful failover service.

3.     Change the host names in the configuration file on each of the primary, backup, and quorum nodes to the new host names. For example, change the host name of a node from cvknode13 to cvk13.

¡     Identify the hostname entries in the /etc/hosts files on the primary, backup, and quorum nodes, and then change the host names to the new ones.

¡     Execute the hostnamectl set-hostname cvk13 command on the target host. If the new host name contains uppercase letters, use the hostnamectl set-hostname --static CVK13 command to make the change.

 

IMPORTANT: If the hostnamectl command is not available in a Ubuntu environment, use the hostname cvknode13/CVK13 command to change the host name. Then, modify the /etc/hostname configuration file.

 

¡     Edit the host names in the stateful failover configuration file at /etc/cmsd/cmsd.xml on each of the primary, backup, and quorum nodes. For example, change name="cvknode13" to name="cvk13".

¡     Edit the /etc/drbd.d/r0.res file to change the host names to the new ones. If the /etc/drbd.d/r1.res file exists, edit it as well. Skip this step on the quorum node. For example, change on cvknode13 to on cvk13.

¡     Execute the service network restart command on the primary and backup nodes to restart the network for the stateful failover system.

4.     Execute the service cmsd start command on the primary, backup, and quorum nodes in turn to start the stateful failover service on these nodes.

Replace advanced quorum with ping quorum

Prerequisites

Make sure the primary and backup nodes can ping the quorum IPs, which are IP addresses of any two external hosts outside the stateful failover system.

Restrictions and guidelines

Before you replace advanced quorum with ping quorum, you must first stop the stateful failover service. This will block access to the CVM platform via its VIP. Schedule this change for off-peak hours to minimize disruptions.

Procedure

1.     Execute the ms_resource_stop.sh command on the primary node to disable the stateful failover service.

2.     Edit the configuration file at /etc/cmsd/cmsd.xml on both the primary and backup nodes. Identify the line that contains the role="3" key-value pair. At this line, replace the values for the ip and ip_other attributes with the quorum IPs for ping quorum, change the value for the type attribute from 1 to 2, and change the host name to cvm-03.

The following is an example:

<host name="cvknode178" role="3" ip="10.99.219.11" priority="3" ip_other="4.4.4.4" type="1"/>

Change it as follows:

<host name="cvm-03" role="3" ip="10.125.32.44" priority="3" ip_other="10.125.62.30" type="2" />

3.     On the original quorum node for advanced quorum, execute the systemctl disable cmsd.service command, and then delete the configuration file at etc/cmsd/cmsd.xml.

4.     Execute the ms_resource_start.sh command on the primary node to start the stateful failover service.

5.     Execute the crm status command to verify that the stateful failover service is operating correctly.