Recover from a failed compute node

admin

Recover from a failed compute node
5 @) V8 U8 N1 [updated: 2018-12-18 05:07
% Y1 y2 s* k0 D" W4 j+ ZRecover from a failed compute node¶
3 j- P" ^3 Z9 p; nIf you deploy Compute with a shared file system, you can use several methods to quickly recover from a node failure. This section discusses manual recovery.

Evacuate instances¶
If a hardware malfunction or other error causes the cloud compute node to fail, you can use the nova evacuate command to evacuate instances. See evacuate instances for more information on using the command.
1 X; ]) s1 r9 V$ Q. @
2 @2 I4 e- I+ I! M& WManual recovery¶
To manually recover a failed compute node:

Identify the VMs on the affected hosts by using a combination of the openstack server list and openstack server show commands or the euca-describe-instances command.
6 X1 n/ ?$ i6 ?' n
For example, this command displays information about the i-000015b9 instance that runs on the np-rcc54 node:

! S4 G4 |6 X6 Z' \$ euca-describe-instances
i-000015b9 at3-ui02 running nectarkey (376, np-rcc54) 0 m1.xxlarge 2012-06-19T00:48:11.000Z 115.146.93.60
Query the Compute database for the status of the host. This example converts an EC2 API instance ID to an OpenStack ID. If you use the nova commands, you can substitute the ID directly. This example output is truncated:
2 T; A  ^, N7 R6 b! F& T' |
mysql> SELECT * FROM instances WHERE id = CONV('15b9', 16, 10) \G;
5 ]6 a% s2 x3 R6 V* q* c, T. {7 p*************************** 1. row ***************************
5 h/ H9 v0 x# E2 a+ V* K8 Pcreated_at: 2012-06-19 00:48:11
updated_at: 2012-07-03 00:35:11
. n5 K3 q0 o2 Y3 [# T: ~3 V$ hdeleted_at: NULL
- u$ K5 n/ I/ R...
id: 5561
...
power_state: 5
vm_state: shutoff
...
& a, N& Z( Q+ S# p' E/ `hostname: at3-ui02
& G& J. n/ J* S* W3 Jhost: np-rcc54
...
uuid: 3f57699a-e773-4650-a443-b4b37eed5a06
2 f1 Y; [/ P/ b& i...
4 r) ]: }$ Y! @8 d0 xtask_state: NULL
. ^5 I# a# ^5 W, t& {( G7 d...
Note
- n" ]6 P! w/ M/ e) I% R0 i
& ?4 l' ~8 ?* {Find the credentials for your database in /etc/nova.conf file.
& c4 \$ y* s0 B4 w; j
  E: k; }+ ]+ k, U9 ODecide to which compute host to move the affected VM. Run this database command to move the VM to that host:

mysql> UPDATE instances SET host = 'np-rcc46' WHERE uuid = '3f57699a-e773-4650-a443-b4b37eed5a06';
4 S! B3 M  I  [& @$ E- ?, Z/ eIf you use a hypervisor that relies on libvirt, such as KVM, update the libvirt.xml file in /var/lib/nova/instances/[instance ID] with these changes:
& R9 W0 a/ C# H
( A4 r$ l$ d. k* cChange the DHCPSERVER value to the host IP address of the new compute host.
Update the VNC IP to 0.0.0.0.
1 E( S! ^6 X9 iReboot the VM:
0 Q5 K$ P" C; X3 `
7 F2 N9 b/ E7 Y( g$ openstack server reboot 3f57699a-e773-4650-a443-b4b37eed5a06
Typically, the database update and openstack server reboot command recover a VM from a failed host. However, if problems persist, try one of these actions:

Use virsh to recreate the network filter configuration.
Restart Compute services.
$ x% d# ?1 r% q  |: IUpdate the vm_state and power_state fields in the Compute database.
Recover from a UID/GID mismatch¶
7 W. \3 H0 v% a0 mSometimes when you run Compute with a shared file system or an automated configuration tool, files on your compute node might use the wrong UID or GID. This UID or GID mismatch can prevent you from running live migrations or starting virtual machines.

+ R2 n* h" r* f1 v: pThis procedure runs on nova-compute hosts, based on the KVM hypervisor:

Set the nova UID to the same number in /etc/passwd on all hosts. For example, set the UID to 112.

+ n: E9 T  S: bNote
6 Z4 [5 J0 L/ p1 ^; `3 A; v
6 V. x. B0 ?5 E- TChoose UIDs or GIDs that are not in use for other users or groups.
) {% a5 m' \5 V# ^5 x4 b, a7 C
Set the libvirt-qemu UID to the same number in the /etc/passwd file on all hosts. For example, set the UID to 119.

Set the nova group to the same number in the /etc/group file on all hosts. For example, set the group to 120.
0 R- u2 y. F1 z
2 N" Y$ I8 s" O! J  m- FSet the libvirtd group to the same number in the /etc/group file on all hosts. For example, set the group to 119.
* D: _3 Z9 f6 s% \6 z  Z2 y
* R- ]) F- _4 J) G: K3 ZStop the services on the compute node.
) U" y8 {( {1 \* M( k, g/ A
Change all files that the nova user or group owns. For example:
; C1 ~# f: D  L2 g
# find / -uid 108 -exec chown nova {} \;
# note the 108 here is the old nova UID before the change
# find / -gid 120 -exec chgrp nova {} \;
Repeat all steps for the libvirt-qemu files, if required.

! j: ~  W! \  {# b+ ?Restart the services.

To verify that all files use the correct IDs, run the find command.
$ h+ R5 U4 Q$ R% A
* N% H, _3 U1 n- J, zRecover cloud after disaster¶
This section describes how to manage your cloud after a disaster and back up persistent storage volumes. Backups are mandatory, even outside of disaster scenarios.

& F/ |, N( ]& C3 x6 N, O! Z9 EFor a definition of a disaster recovery plan (DRP), see https://en.wikipedia.org/wiki/Disaster_Recovery_Plan.
0 u2 I' z7 L9 u/ \: ~# l' \6 H
% X$ c: y9 x1 U' Y1 \& Y6 QA disk crash, network loss, or power failure can affect several components in your cloud architecture. The worst disaster for a cloud is a power loss. A power loss affects these components:
: c  @' r9 o; Q
A cloud controller (nova-api, nova-objectstore, nova-network)
A compute node (nova-compute)
A storage area network (SAN) used by OpenStack Block Storage (cinder-volumes)
/ i9 A3 |& i( A- z% NBefore a power loss:

9 z  F! l0 F4 m/ CCreate an active iSCSI session from the SAN to the cloud controller (used for the cinder-volumes LVM's VG).
Create an active iSCSI session from the cloud controller to the compute node (managed by cinder-volume).
: C! Y% q2 W. m5 f$ m0 [# pCreate an iSCSI session for every volume (so 14 EBS volumes requires 14 iSCSI sessions).
Create iptables or ebtables rules from the cloud controller to the compute node. This allows access from the cloud controller to the running instance.
& ~2 V4 E7 y1 L. f: |Save the current state of the database, the current state of the running instances, and the attached volumes (mount point, volume ID, volume status, etc), at least from the cloud controller to the compute node.
After power resumes and all hardware components restart:
( a; ~1 Z. U$ E8 x, m: i
The iSCSI session from the SAN to the cloud no longer exists.

The iSCSI session from the cloud controller to the compute node no longer exists.
( U6 ~) s# x$ N
nova-network reapplies configurations on boot and, as a result, recreates the iptables and ebtables from the cloud controller to the compute node.
/ \( W! L1 C; o) _$ R
Instances stop running.
% T% V, ?4 R; i. N' X  ~* L* m
* y# J4 `8 ~0 Y3 \1 rInstances are not lost because neither destroy nor terminate ran. The files for the instances remain on the compute node.
7 V8 h9 u( V" ]+ H
$ n- a0 ^; R/ s( D. bThe database does not update.

Begin recovery

Warning
% m( E; Y- x, W
, Z  J5 D8 Z6 Z- M, ADo not add any steps or change the order of steps in this procedure.

Check the current relationship between the volume and its instance, so that you can recreate the attachment.

Use the openstack volume list command to get this information. Note that the openstack client can get volume information from OpenStack Block Storage.
- a3 P: k- _5 p: Y2 z
8 ~' B7 F3 |6 x; ~0 VUpdate the database to clean the stalled state. Do this for every volume by using these queries:
% A& a6 i0 C* o2 J& d
mysql> use cinder;
2 F+ G: y4 U2 a# y2 wmysql> update volumes set mountpoint=NULL;
mysql> update volumes set status="available" where status <>"error_deleting";
mysql> update volumes set attach_status="detached";
mysql> update volumes set instance_id=0;
2 m$ S. g9 W$ ~/ j1 J7 aUse openstack volume list command to list all volumes.
' Q2 x1 M1 v$ r9 H% R
+ B0 z9 j/ j  X% j1 O5 kRestart the instances by using the openstack server reboot INSTANCE command.

Important
) r; {" y  C. t% y) v
Some instances completely reboot and become reachable, while some might stop at the plymouth stage. This is expected behavior. DO NOT reboot a second time.

Instance state at this stage depends on whether you added an /etc/fstab entry for that volume. Images built with the cloud-init package remain in a pending state, while others skip the missing volume and start. You perform this step to ask Compute to reboot every instance so that the stored state is preserved. It does not matter if not all instances come up successfully. For more information about cloud-init, see help.ubuntu.com/community/CloudInit/.
* O( r3 k' t) m; B0 s) k' Z
If required, run the openstack server add volume command to reattach the volumes to their respective instances. This example uses a file of listed volumes to reattach them:
/ t* T- S5 ?, {  _4 d1 m
! F1 E2 ^( _/ d' `#!/bin/bash

& \( e8 U0 d& H4 l8 Twhile read line; do
    volume=`echo $line | $CUT -f 1 -d " "`
0 Z, C7 R* W/ X& |& D0 L    instance=`echo $line | $CUT -f 2 -d " "`
) }: `8 o9 b) w0 E: h4 U    mount_point=`echo $line | $CUT -f 3 -d " "`
        echo "ATTACHING VOLUME FOR INSTANCE - $instance"
( u& O3 L# S* h3 P    openstack server add volume $instance $volume $mount_point
! k! K% z9 ]' t& J    sleep 2
done < $volumes_tmp_file
Instances that were stopped at the plymouth stage now automatically continue booting and start normally. Instances that previously started successfully can now see the volume.

Log in to the instances with SSH and reboot them.
0 O" G7 E$ B1 k" E3 Q2 }
; G  K% @) i; o( |. }% E) M$ lIf some services depend on the volume or if a volume has an entry in fstab, you can now restart the instance. Restart directly from the instance itself and not through nova:
/ R$ U1 ?$ Z) x
# shutdown -r now
4 |1 C7 k; _0 ~' \. D+ W5 SWhen you plan for and complete a disaster recovery, follow these tips:

Use the errors=remount option in the fstab file to prevent data corruption.

In the event of an I/O error, this option prevents writes to the disk. Add this configuration option into the cinder-volume server that performs the iSCSI connection to the SAN and into the instances' fstab files.
Do not add the entry for the SAN's disks to the cinder-volume's fstab file.

- X( y* c/ A3 ~$ d6 dSome systems hang on that step, which means you could lose access to your cloud-controller. To re-run the session manually, run this command before performing the mount:

# iscsiadm -m discovery -t st -p $SAN_IP $ iscsiadm -m node --target-name $IQN -p $SAN_IP -l
6 o* Y0 e$ _, L9 D$ \0 R/ cOn your instances, if you have the whole /home/ directory on the disk, leave a user's directory with the user's bash files and the authorized_keys file instead of emptying the /home/ directory and mapping the disk on it.

* v, S# a. V0 D: g% `This action enables you to connect to the instance without the volume attached, if you allow only connections through public keys.
2 v# Y  N5 N- |( ~
To script the disaster recovery plan (DRP), use the https://github.com/Razique bash script.
, Q" B4 C( X* [% }+ _, y
! F! d0 t) j0 k1 ^5 dThis script completes these steps:

: K  V+ U4 @% @6 J! Q2 N: GCreates an array for instances and their attached volumes.
: L$ y, t3 m7 [- DUpdates the MySQL database.
. j8 p7 H5 C0 @& o: z* X. G& BRestarts all instances with euca2ools.
Reattaches the volumes.
Uses Compute credentials to make an SSH connection into every instance.
The script includes a test mode, which enables you to perform the sequence for only one instance.

) P. ^8 u9 ^2 H, B; tTo reproduce the power loss, connect to the compute node that runs that instance and close the iSCSI session. Do not detach the volume by using the openstack server remove volume command. You must manually close the iSCSI session. This example closes an iSCSI session with the number 15:

& @, ]9 g. l* l7 \, g# iscsiadm -m session -u -r 15
( P3 _: l7 Z) K. H- aDo not forget the -r option. Otherwise, all sessions close.

Warning
1 H" M- K  E7 w: P
There is potential for data loss while running instances during this procedure. If you are using Liberty or earlier, ensure you have the correct patch and set the options appropriately.
4 N: e. [4 f+ x: K/ N" d
updated: 2018-12-18 05:07