In order to balance and ensure the smooth transition from traditional application systems to cloud computing, OpenStack community does not provide a complete solution for high availability of virtual machines, but it also provides some working mechanisms with external monitoring services so that users can realize high availability of virtual machines themselves. In addition, in the Liberty version, OpenStack implements and improves the related NovaAPI interface, so as to better cooperate with the external high-availability system to monitor the change of the service status of Nova and failover the virtual machine. For example, Red Hat's RDO realizes high availability of virtual machines based on Pacemaker-remote, and the community also monitors nova-compute services based on Zookeeper to realize high availability of virtual machines.
In OpenStack, the so-called virtual machine high availability means that when a physical computing node has a hardware failure (such as disk damage, downtime caused by CPU or memory failure, physical network failure, power failure, etc.), the node is automatically shut down, and the virtual machine on the node is restarted on other healthy computing nodes in the cluster. If virtual machines can be migrated dynamically, this is the best high availability scheme. In the high availability scheme of OpenStack virtual machine, although the software used is different, it is usually based on three steps: monitoring, isolation and recovery.
1) The purpose of monitoring is to judge whether the Hypervisor is faulty or not, and provide the execution basis for the isolation operation. The monitoring function consists of two parts, one is to detect the host failure, and the other is to trigger the automatic response task (isolation and recovery) after the host failure. There has always been a debate in the community about whether the monitoring function should be integrated into Nova: it is believed that the monitoring of computing node services should be integrated into Nova project, mainly because Nova services have obtained the health status of the infrastructure on which their operation depends to some extent, or Nova itself can track active computing nodes. However, the tracking and monitoring of computing nodes by Nova can only detect the failure of Nova-compute service, and the failure of Nova-compute service does not mean that the virtual machine also fails, that is, whether the Nova-compute service of computing nodes is normal or not is not necessarily related to the failure of the virtual machine on the node. Some communities also suggest that monitoring function should be integrated into the Heat project, but this scheme requires OpenStack end users to restart the virtual machine with the Heat template when the virtual machine fails (but this should be the task of the cloud administrator, not the user's execution). In the current OpenStack high availability environment, Pacemaker combined with Corosync is the most widely used service high availability monitoring tool. However, due to historical reasons, the number of computing nodes supported by Corosync is limited, and the Pacemaker_remote proposed by Redhat solves this limitation.
2) Isolation is a key operation in high availability cluster, and the so-called cluster "brain crack" is usually caused by imperfect isolation operation. In the high availability of Nova service in OpenStack cluster, isolation is to completely isolate the failed computing node from the cluster and make it an isolated node. In the instance high availability environment, the compute nodes may fail for various reasons. Before restarting the virtual machine of the failed computing node on other healthy nodes, it must be ensured that the virtual machine does not exist, otherwise two identical virtual machines may appear in an OpenStack cluster at the same time. To make matters worse, if virtual machines are deployed on * * * shared storage, two identical virtual machines will run at the same time. Writing a copy of data by two virtual machines usually leads to data corruption and two identical IP addresses in the same network. Therefore, before the high-availability software recovers the failed virtual machine, the monitor must isolate the failed computing node, otherwise it will inevitably cause various damages to the virtual machine. Pacemaker provides isolation function for cluster nodes. If you use other clustering tools, you need to implement the isolation function yourself.
3) After monitoring the failure of the compute node and isolating the node, the user's virtual machine can be restored. In Nova, the function of virtual machine recovery is mainly the Evacuate command provided by Nova. When Evacuate is called, the virtual machine on the failed computing node will be automatically evacuated and resumed on the new node. In order to restore virtual machines, virtual machines should be created on * * * shared storage. Alternatively, you can create a virtual machine on a volume provided by Cinder. However, * * * Shared storage or volumes is not a prerequisite for the successful implementation of virtual machines. If the virtual machine is not created by the above two schemes, the virtual machine will be restored on other nodes, but this is a lossy restoration (because the virtual machine only recreates the same virtual machine on other nodes with the same basic image of the original virtual machine, and the data changed from the basic image in the original virtual machine cannot be restored).
At present, OpenStack has no complete monitoring, isolation and recovery scheme. Therefore, users must realize service monitoring and node isolation by themselves, and at the same time trigger evacuation operation on the fault computing node. If you use Pacemaker Cluster Explorer, you need to implement an evacuation resource proxy on the computing node, which allows Pacemaker to trigger evacuation operations on the node.
Nova provides the Evacuate API to restore the virtual machine on the failed computing node, which is the basis for achieving high availability of the virtual machine on the computing node. After monitoring and isolating the failure of the compute node, evacuation operation can be triggered to restore the virtual machine on the node. In essence, Evacuate is an extension of Rebuild function, or it is based on the requirements of virtual machine HA, but Rebuild function still has its practicability. The main difference between Rebuild and Evacuate is that Rebuild will refresh the virtual machine mirror disk, that is, re-create a virtual machine with the same ID with a new image, so Rebuild does not need to enjoy storage, and its function is more like reinstalling another operating system with the same hardware (for example, replacing windows system with Linux system); Evacuation is the real recovery, including system and user data. In addition, migration and reconstruction only support virtual machines in active and stopped states, but not in suspended, suspended and closed states.
The specific operation flow of Evacuate depends on the configuration of virtual machine and the underlying storage architecture. If the virtual machine is based on the "transient" storage of the local file system of the computing node, the evacuation operation is created in other nodes with the same image as the original virtual machine, and the new and old virtual machines have the same configuration, such as the same instance ID, image ID, attached volume, style, IP, etc. If the virtual machine is located on * * * shared storage, Evacuate will restart the virtual machine on other nodes based on the same virtual machine file and keep the configuration of the virtual machine unchanged. If the virtual machine is based on the back end of a volume, Evacuate will rebuild the virtual machine and start it from the same volume. Therefore, virtual machines based on * * * shared storage and volume backend can be restored as they are, while virtual machines based on local temporary storage cannot restore user data (data located on temporary storage).
1) Check the virtual machine information before rebuilding (nova show? Vmid). Record UUID, IP address, host name and volume installation.
2) Perform the reconstruction operation. Here, the original image cirros-image-name is still used to rebuild vm-name, and users can choose any other image to rebuild it. Nova? Reconstruction? Virtual machine name cirros image name
3) Check 3) Reconstructed virtual machine information (nova display? Vmid). Observe whether the host name, UUID, IP address and volume mount have changed. Under normal circumstances, these parameters should remain unchanged after reconstruction.
1) Save user data in the virtual machine operating system before evacuation for verification after evacuation.
[root@nfs-server ~]#？ Echo "The data should be recovered after evacuation" >; ? /root/test.txt
2) check the virtual machine information before evacuation (nova show? Vmid). Verify the host of the virtual machine and confirm the target host of the evacuation operation. Evacuation requires the use of * * * shared storage or volume-based virtual machines, where the nfs server is an NFS***-based virtual machine. Nova? Hypervisor-Server Computing 2
3) When the computing node is running normally, the evacuation operation is executed. The evacuation operation requires that the computing service of the host where the virtual machine is located cannot be active, otherwise it is not allowed to be executed. //compute 1 node is running normally, and evacuation is not allowed.
[root @ controller1~] # nova exits nfs _ server compute2.
Mistakes? (BadRequest): calculate? Service? Yes? compute 1？ Is it? Or? Are you online? Use. ? (HTTP？ 400)
4) Perform evacuation operation when calculating node failure. Compute 1 is used to simulate shutdown failure and observe the change of virtual machine state during evacuation.
//Turn off the computer 1 node
[root@compute 1~]# Shut down
//evacuate is executed, and the target node is compute2.
[root @ controller1~] # nova exits nfs _ server compute2.
5) Check the change of the virtual machine after evacuation (nova show vmid). The host where the virtual machine is located should drift from compute 1 to compute2.
6) Verify whether user data is available after evacuation. There are two ways to verify this: change the root password of the virtual machine before evacuation, and now check whether the root password is still available; Before evacuation, user data is stored in/root/test.txt. Now check if the data still exists.
[root@nfs-server ~]#？ more /root/test.txt
This data should be recovered after evacuation.
7) Start the compute 1 node and check whether the virtual machine on the compute 1 node will automatically recover. Under normal circumstances, the evacuated virtual machine will not be restored on the original compute node, and the relevant information of the virtual machine will be automatically cleared during the restoration of the original compute node.
The public cloud depends on the high availability of the application itself, which tolerates the downtime of the physical machine to a certain extent, while the private cloud depends on the high availability of the server. In order to ensure the smooth transition from traditional application system to cloud computing, some working mechanisms with external monitoring services are provided so that users can realize the high availability of virtual machines themselves. Virtual machine high availability is usually realized based on three steps: monitoring, isolation and recovery. The tracking and monitoring of computing nodes by Nova is isolated by detecting the failure of Nova-compute service. Pacemaker provides the isolation function of cluster nodes, and an evacuation resource agent needs to be implemented on the computing nodes, thus allowing Pacemaker to trigger evacuation recovery operations on the nodes. Pacemaker and Corosync are the most widely used high availability service monitoring tools, but Corosync supports a limited number of computing nodes. In addition, Pacemaker_remote proposed by Redhat solves this limitation. To know how to solve this problem, please break it down next time.