Cheap HA KVM Cluster w/ Shared File System

From Labrats.us
Jump to navigationJump to search

Document to go through setting up KVM install on CentOS with clustering.

Network Configuration

Configure hostname

# hostnamectl set-hostname centos-vm1.home.labrats.us

Configure Layer 3 Interface Bonding and Bridge Interface

For this documentation, we are going to use 2 ethernet interfaces, one on the system board (em1) and one on an expansion card (p1p1).

Configure em1

/etc/sysconfig/network-scripts/ipcfg-em1. Make sure the MAC address (HWADDR) is correct.

DEVICE=em1
NAME=em1
UUID=a401d981-52cd-4c4d-8c0d-3f81f182fe45
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes

Configure p1p1

/etc/sysconfig/network-scripts/ipcfg-p1p1. Make sure the MAC address (HWADDR) is correct.

DEVICE=p1p1
NAME=p1p1
UUID=75ba61cf-0781-4023-b451-911f2e0b69d3
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes

Configure bond0

/etc/sysconfig/network-scripts/ipcfg-bond0.

DEVICE=bond0
NAME=bond0
TYPE=Bond
BOOTPROTO=none
#DEFROUTE=yes
#PEERDNS=yes
#PEERROUTES=yes
#IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_PEERDNS=yes
#IPV6_PEERROUTES=yes
#IPV6_FAILURE_FATAL=no
ONBOOT=yes
BRIDGE=virbr0

/etc/modprobe.d/bond0.conf

alias bond0 bonding
options bond0 primary=em1 miimon=100 mode=1 updelay=30000

Configure virbr0

/etc/sysconfig/network-scripts/ipcfg-virbr0.

DEVICE="virbr0"
TYPE=BRIDGE
ONBOOT=yes
BOOTPROTO=none
IPADDR="192.168.1.201"
NETMASK="255.255.255.0"
GATEWAY="192.168.1.1"
DNS1="216.136.95.2"
DNS2="64.132.94.250"
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
NM_CONTROLLED="no"

Enable IP Bridging:

# systemctl start libvirtd
# systemctl enable libvirtd
# echo "net.ipv4.ip_forward = 1" | tee /etc/sysctl.d/99-ipforward.conf
# sudo sysctl -p /etc/sysctl.d/99-ipforward.conf

Configure resolv.conf

domain home.labrats.us
search home.labrats.us labrats.us
nameserver 216.136.95.2
nameserver 64.132.94.250

Disable Network Manager

Disable Network Manager

/bin/systemctl disable NetworkManager
/bin/systemctl disable NetworkManager-dispatcher

Delete Network Manager Packages

# yum erase NetworkManager-tui NetworkManager-glib NetworkManager

Restart networking

# /etc/init.d/network restart
Restarting network (via systemctl):                        [  OK  ]

It is possible that you will need to reboot the server.

# shutdown -r now

Show Interface Bonding

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: p1p1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 30000
Down Delay (ms): 0

Slave Interface: em1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Slave queue ID: 0

Slave Interface: p1p1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Slave queue ID: 0

Setup Bond1 and Bond2

Use similar configuration steps as above to map em2/p1p2 to bond1, and em3/p1p3 to bond3. We will skip the bridging setup for these interfaces, as they are for Cluster and Storage Networks.

Bond interface configurations look like this:

Bond1 interface

/etc/modprobe.d/bond1.conf

alias bond1 bonding
options bond1 primary=em2 miimon=100 mode=1 updelay=30000

/etc/sysconfig/network-scripts/ifcfg-em2

NAME=em2
UUID=903d0880-7c69-4bfd-b928-35f507be0d73
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond1
SLAVE=yes

/etc/sysconfig/network-scripts/ifcfg-p1p2

DEVICE=p1p2
NAME=p1p2
UUID=7516861d-5dc2-4cd7-a4b0-525bb398d8f9
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond1
SLAVE=yes

/etc/sysconfig/network-scripts/ifcfg-bond1

DEVICE=bond1
NAME=bond1
TYPE=Bond
BOOTPROTO=none
IPADDR="192.168.101.201"
NETMASK="255.255.255.0"
NM_CONTROLLED="no"

Bond2 interface

/etc/modprobe.d/bond2.conf

alias bond2 bonding
options bond2 primary=em3 miimon=100 mode=1 updelay=30000

/etc/sysconfig/network-scripts/ifcfg-em3

DEVICE=em3
NAME=em3
UUID=2215a365-2d29-45c1-8f68-fbee29713c86
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond2
SLAVE=yes

/etc/sysconfig/network-scripts/ifcfg-p1p3

DEVICE=p1p3
NAME=p1p3
UUID=7aff546a-1dec-4763-b2fc-8cfcfdaf086b
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond2
SLAVE=yes

/etc/sysconfig/network-interfaces/ifcfg-bond2

DEVICE=bond2
NAME=bond2
TYPE=Bond
BOOTPROTO=none
IPADDR="192.168.102.201"
NETMASK="255.255.255.0"
NM_CONTROLLED="no"

Setup /etc/hosts file

Run the following command to setup /etc/hosts file.

# cat << EOF >> /etc/hosts
192.168.1.200    centos-vm.home.labrats.us centos-vm
192.168.1.201    centos-vm1.home.labrats.us centos-vm1
192.168.1.202    centos-vm2.home.labrats.us centos-vm2
192.168.1.211    centos-vm1-ipmi.home.labrats.us centos-vm1-ipmi
192.168.1.212    centos-vm2-ipmi.home.labrats.us centos-vm2-ipmi
192.168.101.201  centos-vm1-cn.home.labrats.us centos-vm1-cn
192.168.101.202  centos-vm2-cn.home.labrats.us centos-vm2-cn
192.168.102.201  centos-vm1-sn.home.labrats.us centos-vm1-sn
192.168.102.202  centos-vm2-sn.home.labrats.us centos-vm2-sn

EOF

Configure Networking On Each Additional Node

Setup Gluster

Create Gluster Brick

mkfs.xfs -i size=512 /dev/sdb1
mkdir -p /data/brick1
vi /etc/fstab

Mount Brick

Add to /etc/fstab

/dev/sdb1 /data/brick1 xfs defaults 1 2

Mount the brick

# mount /data/brick1

Install Gluster

Install EPEL and Gluster Repos

We'll need to install the Gluster repo for the server package, and EPEL repo to satisfy dependencies.

# rpm -i https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 
# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo

Install glusterfs-server

Now we can install glusterfs-server package.

# yum install glusterfs-server

Start Gluster Server

# service glusterd start
Redirecting to /bin/systemctl start  glusterd.service

Check the status of the gluster service

# service glusterd status
Redirecting to /bin/systemctl status  glusterd.service
glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; disabled)
   Active: active (running) since Wed 2015-09-09 12:27:58 MDT; 12s ago
  Process: 13774 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS)
 Main PID: 13775 (glusterd)
   CGroup: /system.slice/glusterd.service
           └─13775 /usr/sbin/glusterd -p /var/run/glusterd.pid

Sep 09 12:27:58 centos-vm1.home.labrats.us systemd[1]: Started GlusterFS, a clustered file-system server.

Add Firewall Rules

We'll use firewall-cmd in this example. IPTables would have similar rules.

First we have to open up the port for the Gluster service. This runs on TCP port 24007.

# firewall-cmd --add-port=24007/tcp
# firewall-cmd --permanent --add-port=24007/tcp

Next we have to open up a port for each brick, starting at 49152 (for GlusterFS 3.4 and later, 24009 for GlusterFS 3.3 and older). Since we are running two bricks in each node, we only need to listen to 49152 and 49153.

# firewall-cmd --add-port=49152-49153/tcp
# firewall-cmd --permanent --add-port=49152-49153/tcp

If we want to open up GlusterFS to external nodes, we will also need to open up TCP ports 38465, 38466, 38468, 38469 and 2049.

Let's validate that is shows up:

# firewall-cmd --list-all
public (default, active)
  interfaces: virbr0
  sources: 
  services: dhcpv6-client ssh
  ports: 24007/tcp 49152-49153/tcp
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules: 
	

Configure each additional node

Configure each node in the Gluster Cluster with the same configuration.'

Configure Gluster Service

Probe each server from the other one

Node 1:

# gluster peer probe centos-vm2-sn

Node 2:

# gluster peer probe centos-vm1-sn

Create volume directory on each node

This needs to be ran on each node.

# mkdir /data/brick1/gv0

Create Gluster Volume

This only needs to be ran on ONE node.

# gluster volume create gv0 replica 2 centos-vm1-sn:/data/brick1/gv0 centos-vm2-sn:/data/brick1/gv0
volume create: gv0: success: please start the volume to access data

Start Gluster Volume

Again, this only needs to be done on one node.

# gluster volume start gv0
volume start: gv0: success

Check Gluster Volume Status

This can be ran on either node, and should be ran on each to validate operation.

# gluster volume info
 
Volume Name: gv0
Type: Replicate
Volume ID: 877e2a93-89c5-4f19-b98c-72d79abbae83
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: centos-vm1-sn:/data/brick1/gv0
Brick2: centos-vm2-sn:/data/brick1/gv0
Options Reconfigured:
performance.readdir-ahead: on

Set Gluster Service to start with the system

Run this command on each node.

# systemctl enable glusterd.service

Testing Gluster

Perform the following steps to test and validate that Gluster is operating correctly.

Mount Gluster volume on each server

On each node, mount the local Gluster volume.

# mount -t glusterfs localhost:/gv0 /mnt

Test Write & Synchronization

On node 1, run the following commands.

# for i in `seq -w 1 100`; do cp -rp /var/log/messages /mnt/copy-test-$i; done
<pre>

On node 1, check to see that this has been written to the local brick.

<pre>
# ls -lA /data/brick1/gv0 | wc -l

There should be 100 files on the local brick.

On node 2, check both the /mnt directory, and the local brick.

# ls -lA /mnt | wc -l
# ls -lA /data/brick1/gv0 | wc -l

Both should list 100 files. If the numbers are different, check that synchronization is working correctly.

Setup Second Gluster Volume

We will also need to setup a "gv1" volume as well, using the above steps.

Setup KVM

Install KVM

# yum -y install kvm virt-manager libvirt virt-install qemu-kvm
# yum -y install xauth xorg-x11-apps

Mount GlusterFS

# mkdir -p /var/lib/libvirt/images /var/lib/libvirt/configs
# mount -t glusterfs localhost:/gv0 /var/lib/libvirt/images
# mount -t glusterfs localhost:/gv1 /var/lib/libvirt/configs

Start KVM (libvirtd)

We must start libvirtd temporarily to create the virtual machine disk images and config files.

# systemctl start libvirtd.service

Create Virtual Machines

We will create two machines, test-centos-1 and test-centos-2.

# virt-install --connect qemu:///system -n test-centos-1 -r 2048 --vcpus=1 \
  --disk path=/var/lib/libvirt/images/test-centos-1.img,size=10 --graphics vnc,listen=0.0.0.0 \
  --noautoconsole --os-type linux --os-variant rhel7 --accelerate --network=bridge:virbr0 --hvm \
  --cdrom /var/lib/libvirt/images/CentOS-7.0-1406-x86_64-DVD.iso
# virt-install --connect qemu:///system -n test-centos-2 -r 2048 --vcpus=1 \
  --disk path=/var/lib/libvirt/images/test-centos-2.img,size=10 --graphics vnc,listen=0.0.0.0 \
  --noautoconsole --os-type linux --os-variant rhel7 --accelerate --network=bridge:virbr0 --hvm \
  --cdrom /var/lib/libvirt/images/CentOS-7.0-1406-x86_64-DVD.iso

Connect to VNC console and install the OD from CD image. This can be done via a VNC client application, or via the virt-manager GUI application.

Configure Virtual Machines

Configure Guest Network

Remove Network Manager software.

# /bin/systemctl disable NetworkManager
# /bin/systemctl disable NetworkManager-dispatcher
# yum erase NetworkManager-tui NetworkManager-glib NetworkManager

Run the commands below to set up a static ip address (192.168.1.221) and hostname (test-centos-1).

# export remote_hostname=guest1
# export remote_ip=192.168.1.221
# export remote_gateway=192.168.1.1

# hostnamectl set-hostname $remote_hostname

# sed -i.bak "s/.*BOOTPROTO=.*/BOOTPROTO=none/g" /etc/sysconfig/network-scripts/ifcfg-eth0

# cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-eth0
IPADDR0=$remote_ip
PREFIX0=24
GATEWAY0=$remote_gateway
DNS1="216.136.95.2"
DNS2="64.132.94.250"
NM_CONTROLLED="no"
EOF

# systemctl restart network
# systemctl enable network.service
# systemctl enable sshd
# systemctl start sshd

# echo "checking connectivity"
# ping www.google.com

To simplify the tutorial we'll go ahead and disable selinux on the guest. We'll also need to poke a hole through the firewall on port 3121 (the default port for pacemaker_remote) so the host can contact the guest.

# setenforce 0
# # sed -i.bak "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config
# firewall-cmd --add-port 3121/tcp --permanent
# firewall-cmd --add-port 3121/tcp

At this point you should be able to ssh into the guest from the host.

Configure pacemaker_remote

On the 'host' machine, run these commands to generate an authkey and copy it to the /etc/pacemaker folder on both the host and guest.

# mkdir -p --mode=0750 /etc/pacemaker
# chgrp haclient /etc/pacemaker
# dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1
# scp -r /etc/pacemaker root@192.168.1.221:/etc/

Now on the 'guest', install the pacemaker-remote package, and enable the daemon to run at startup. In the commands below, you will notice the pacemaker package is also installed. It is not required; the only reason it is being installed for this tutorial is because it contains the Dummy resource agent that we will use later for testing.

# yum install -y pacemaker pacemaker-remote resource-agents
# systemctl enable pacemaker_remote.service

Now start pacemaker_remote on the guest and verify the start was successful.

# systemctl start pacemaker_remote.service

# systemctl status pacemaker_remote

  pacemaker_remote.service - Pacemaker Remote Service
	  Loaded: loaded (/usr/lib/systemd/system/pacemaker_remote.service; enabled)
	  Active: active (running) since Thu 2013-03-14 18:24:04 EDT; 2min 8s ago
	Main PID: 1233 (pacemaker_remot)
	  CGroup: name=systemd:/system/pacemaker_remote.service
		  └─1233 /usr/sbin/pacemaker_remoted

  Mar 14 18:24:04 guest1 systemd[1]: Starting Pacemaker Remote Service...
  Mar 14 18:24:04 guest1 systemd[1]: Started Pacemaker Remote Service.
  Mar 14 18:24:04 guest1 pacemaker_remoted[1233]: notice: lrmd_init_remote_tls_server: Starting a tls listener on port 3121.

Verify Host Connection to Guest

Before moving forward, it's worth verifying that the host can contact the guest on port 3121. Here's a trick you can use. Connect using ssh from the host. The connection will get destroyed, but how it is destroyed tells you whether it worked or not.

First add guest1 to the host machine's /etc/hosts file if you haven't already. This is required unless you have dns setup in a way where guest1's address can be discovered.

# cat << EOF >> /etc/hosts
192.168.1.221     test-centos-1
EOF

If running the ssh command on one of the cluster nodes results in this output before disconnecting, the connection works.

# ssh -p 3121 test-centos-1
ssh_exchange_identification: read: Connection reset by peer

If you see this, the connection is not working.

# ssh -p 3121 test-centos-1
ssh: connect to host test-centos-1 port 3121: No route to host

Repeat for second (and additional) guest virtual machines.

Shut Down Virtual Machines

Power down guest virtual machines, as they will be controlled by Pacemaker, as documented below.

From the host, youcan run this command.

# virsh shutdown test-centos-1
# virsh shutdown test-centos-2

Export Virtual Machine Config Files

We will export the config files for the two virtual machines. This will later be used for loading into Pacemaker.

# virsh dumpxml test-centos-1 > /var/lib/libvirt/configs/test-centos-1.xml
# virsh dumpxml test-centos-2 > /var/lib/libvirt/configs/test-centos-2.xml

Shut Down KVM (libvirt)

We can shut down KVM, as it will be started by Pacemaker following the below steps.

# systemctl start libvirtd.service
# systemctl disable libvirtd.service

Unmount Gluster File Systems

We need to unmount the Gluster file systems, as we will be mounting them with pacemaker below.

# umount /var/lib/libvirt/images
# umount /var/lib/libvirt/configs

Setup Pacemaker Cluster

Pacemaker Firewall rules

Add the following rules for pacemaker/pcs/corosync to communicate correctly.

# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --add-service=high-availability
# firewall-cmd --permanent --direct --add-rule ipv4 filter IN_public_allow 0 -p igmp -j ACCEPT
# firewall-cmd --direct --add-rule ipv4 filter IN_public_allow 0 -p igmp -j ACCEPT
# firewall-cmd --permanent --add-port=5405/tcp
# firewall-cmd --add-port=5405/tcp

Disable SELinux

# sed -i.bak "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config

Install Pacemaker Packages

# yum install pacemaker corosync pcs resource-agents

Setup /etc/hosts file

Create the /etc/hosts file containing entries for each hypervisor host in the cluster.

# cat << EOF >> /etc/hosts
192.168.1.201     centos-vm1
192.168.1.202     centos-vm2
EOF

Setup Cluster Auth

Run the following command on ALL cluster to setup user authentication.

# passwd hacluster

Run the following on ONE node to authenticate the cluster nodes/

# pcs cluster auth centos-vm1-cn centos-vm2-cn

Setup Cluster

Run this on ALL hypervisor machines.

# pcs cluster setup --local --name mycluster centos-vm1-cn centos-vm2-cn

Start Cluster Software

Start the cluster. Run the following on ONE node.

# pcs cluster start --all

Verify Cluster Operation

Verify corosync membership

# pcs status corosync

Membership information
----------------------
    Nodeid      Votes Name
         1          1 centos-vm1.home.labrats.us (local)
         2          1 centos-vm2.home.labrats.us

Verify pacemaker status. It may take a moment or two for the other node to appear.

# pcs status
Cluster name: mycluster
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Tue Sep 15 14:16:51 2015		Last change: Tue Sep 15 14:14:01 2015 by hacluster via crmd on centos-vm2
Stack: corosync
Current DC: centos-vm2 (version 1.1.13-a14efad) - partition with quorum
2 nodes and 0 resources configured

Online: [ centos-vm1 centos-vm2 ]

Full list of resources:


PCSD Status:
  centos-vm1: Offline
  centos-vm2: Offline

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: inactive/disabled

Verify Corosync relationship.

# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
	id	= 192.168.101.201
	status	= ring 0 active with no faults

Setup Pacemaker Active/Standby Cluster

We will initiall setup an active/standby cluster to get things running and tested. This only works in a two-node cluster.

Setup PCSD and Virtual IP

Start PCSD on Primary Node

We need to start PCSD on the Promary node, so it can generate the certificate and key files.

# systemctl start pcsd.service
# systemctl status pcsd.service
pcsd.service - PCS GUI and remote configuration interface
   Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled)
   Active: active (running) since Tue 2015-09-15 13:54:50 MDT; 7s ago
 Main PID: 19779 (pcsd)
   CGroup: /system.slice/pcsd.service
           ├─19779 /bin/sh /usr/lib/pcsd/pcsd start
           ├─19783 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb
           └─19784 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb

Sep 15 13:54:50 centos-vm1-cn systemd[1]: Started PCS GUI and remote configuration interface.

Copy PCSD certificate and key files to other node

We need to make sure that both nodes have the same certificate and key files, or our browser will pester us each time the service fails over.

# cd /var/lib/pcsd
# scp -pv pcsd.* root@centos-vm2-cn:/var/lib/pcsd

Start PCSD opn Secondary Node

# systemctl start pcsd.service
# systemctl status pcsd.service
pcsd.service - PCS GUI and remote configuration interface
   Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled)
   Active: active (running) since Tue 2015-09-15 14:01:24 MDT; 3s ago
 Main PID: 21311 (pcsd)
   CGroup: /system.slice/pcsd.service
           ├─21311 /bin/sh /usr/lib/pcsd/pcsd start
           ├─21315 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb
           ├─21316 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb
           └─21319 python /usr/lib/pcsd/systemd-notify-fix.py

Sep 15 14:01:24 centos-vm2-cn systemd[1]: Started PCS GUI and remote configuration interface.

Create Virtual IP Resource

Now we create the virtual IP resource.

First we will make a backup of the current cluster config.

# cd /tmp
# pcs cluster cib /tmp/cluster-active_config.orig

Now we will configure the virtual IP in an offline file.

# pcs cluster cib /tmp/VirtualIP_cfg

# pcs -f /tmp/VirtualIP_cfg resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.1.200 cidr_netmask=24 op monitor interval=30
# pcs -f /tmp/VirtualIP_cfg constraint location ClusterIP prefers centos-vm1-cn=200
# pcs -f /tmp/VirtualIP_cfg constraint location ClusterIP prefers centos-vm2-cn=50

Now we will push the configuration to the cluster.

# pcs cluster cib-push /tmp/VirtualIP_cfg
CIB updated

Now we can whow the status of the Virtual IP.

# pcs status
Cluster name: mycluster
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Tue Sep 15 14:26:37 2015		Last change: Tue Sep 15 14:25:45 2015 by root via cibadmin on centos-vm1-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
2 nodes and 1 resource configured

Online: [ centos-vm1-cn centos-vm2-cn ]

Full list of resources:

 ClusterIP	(ocf::heartbeat:IPaddr2):	Stopped

PCSD Status:
  centos-vm1-cn: Online
  centos-vm2-cn: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/disabled

Note that the service shows "stopped". This is because we have not configured or disabled fencing yet.

Disable Fencing

To disable fencing, run the following commands. We will discuss enabling it later.

# pcs property set stonith-enabled=false

Now we can check the status again, to see if the new service is running.

# pcs status
Cluster name: mycluster
Last updated: Tue Sep 15 14:28:59 2015		Last change: Tue Sep 15 14:28:55 2015 by root via cibadmin on centos-vm1-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
2 nodes and 1 resource configured

Online: [ centos-vm1-cn centos-vm2-cn ]

Full list of resources:

 ClusterIP	(ocf::heartbeat:IPaddr2):	Started centos-vm1-cn

PCSD Status:
  centos-vm1-cn: Online
  centos-vm2-cn: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/disabled

Enable PCSD at system startup

We can tell the system to start pcsd at startup by running the below on ALL nodes.

# systemctl enable pcsd.service

Setup GlusterFS and Libvirtd (PCS)

Gluster setup is basically to control the mounted of the Gluster File system after system boot. This is needed as the Gluster application is not running when the system mounted the normal file systems, so can not be mounted at the same time.

To turn up GlusterFS mounted, running the following commands on an offline configuration file.

# pcs cluster cib /tmp/GlusterFS_cfg

# pcs -f /tmp/GlusterFS_cfg resource create gluster-configs Filesystem device="localhost:/gv1" directory="/var/lib/libvirt/configs/" fstype="glusterfs"
# pcs -f /tmp/GlusterFS_cfg resource create gluster-images Filesystem device="localhost:/gv0" directory="/var/lib/libvirt/images/" fstype="glusterfs"
# pcs -f /tmp/GlusterFS_cfg resource create libvirtd systemd:libvirtd
# pcs -f /tmp/GlusterFS_cfg constraint colocation add gluster-configs with gluster-images INFINITY
# pcs -f /tmp/GlusterFS_cfg constraint colocation add libvirtd with gluster-images INFINITY
# pcs -f /tmp/GlusterFS_cfg constraint order set gluster-configs gluster-images libvirtd sequential=true require-all=true setoptions kind=Serialize symmetrical=true

Now we can push the GlusterFS configs to the cluster.

# pcs cluster cib-push /tmp/GlusterFS_cfg
CIB updated

Let's show the cluster status.

# pcs status
Cluster name: mycluster
Last updated: Tue Sep 15 14:51:03 2015		Last change: Tue Sep 15 14:50:17 2015 by root via cibadmin on centos-vm1-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
2 nodes and 4 resources configured

Online: [ centos-vm1-cn centos-vm2-cn ]

Full list of resources:

 ClusterIP	(ocf::heartbeat:IPaddr2):	Started centos-vm1-cn
 gluster-configs	(ocf::heartbeat:Filesystem):	Started centos-vm2-cn
 gluster-images	(ocf::heartbeat:Filesystem):	Started centos-vm2-cn
 libvirtd	(systemd:libvirtd):	Started centos-vm2-cn

PCSD Status:
  centos-vm1-cn: Online
  centos-vm2-cn: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Setup KVM (PCS)

We actually handled this in the previous section, setting the libvirtd resource to run co-located with the GlusterFS resources. We also serialized the order to ensure that GlusterFS resource was running and completed start up before libvirtd resource is attempted to start.

Setup Virtual Machines (PCS)

To turn up Virtual Machine resources, running the following commands on an offline configuration file.

# pcs cluster cib /tmp/virtual-machine_cfg

# pcs -f /tmp/virtual-machine_cfg resource create test-centos-1_vm VirtualDomain hypervisor="qemu:///system" config="/var/lib/libvirt/configs/test-centos-1.xml" meta remote-node=test-centos-1
# pcs -f /tmp/virtual-machine_cfg resource create test-centos-2_vm VirtualDomain hypervisor="qemu:///system" config="/var/lib/libvirt/configs/test-centos-2.xml" meta remote-node=test-centos-2
# pcs -f /tmp/virtual-machine_cfg constraint colocation add test-centos-1_vm with libvirtd INFINITY
# pcs -f /tmp/virtual-machine_cfg constraint colocation add test-centos-2_vm with libvirtd INFINITY
# pcs -f /tmp/virtual-machine_cfg constraint order start libvirtd then start test-centos-1_vm kind=Serialize symmetrical=true
# pcs -f /tmp/virtual-machine_cfg constraint order start libvirtd then start test-centos-2_vm kind=Serialize symmetrical=true

We also want to tell teh cluster not to attempt to run the GlusterFS on teh virtual machines.

# pcs -f /tmp/virtual-machine_cfg constraint location gluster-configs avoids test-centos-1
# pcs -f /tmp/virtual-machine_cfg constraint location gluster-images avoids test-centos-1
# pcs -f /tmp/virtual-machine_cfg constraint location gluster-configs avoids test-centos-2
# pcs -f /tmp/virtual-machine_cfg constraint location gluster-images avoids test-centos-2

Now we can push the Virtual Machine configs to the cluster.

# pcs cluster cib-push /tmp/virtual-machine_cfg
CIB updated

Let's show the cluster status.

# pcs status
Cluster name: mycluster
Last updated: Tue Sep 15 15:01:23 2015		Last change: Tue Sep 15 15:01:00 2015 by root via crm_resource on centos-vm2-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
4 nodes and 8 resources configured

Online: [ centos-vm1-cn centos-vm2-cn ]
GuestOnline: [ test-centos-1@centos-vm2-cn test-centos-2@centos-vm2-cn ]

Full list of resources:

 ClusterIP	(ocf::heartbeat:IPaddr2):	Started centos-vm1-cn
 gluster-configs	(ocf::heartbeat:Filesystem):	Started centos-vm2-cn
 gluster-images	(ocf::heartbeat:Filesystem):	Started centos-vm2-cn
 libvirtd	(systemd:libvirtd):	Started centos-vm2-cn
 test-centos-1_vm	(ocf::heartbeat:VirtualDomain):	Started centos-vm2-cn
 test-centos-2_vm	(ocf::heartbeat:VirtualDomain):	Started centos-vm2-cn

PCSD Status:
  centos-vm1-cn: Online
  centos-vm2-cn: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Setup FAKE Services (PCS)

We want to run fake services on the virtual machines to test out the pacemaker-remote service. This is not a critical part of the cluster, but it can come in handy if we are trying to track cluster services, such as DNS, mail, etc.

To turn up FAKE service resource, running the following commands on an offline configuration file.

# pcs cluster cib /tmp/FAKE_cfg

# pcs -f /tmp/FAKE_cfg resource create FAKE1 ocf:pacemaker:Dummy
# pcs -f /tmp/FAKE_cfg resource create FAKE2 ocf:pacemaker:Dummy

We will also set preference for the FAKE resources, so they only run on the virtual guests.

# pcs -f /tmp/FAKE_cfg constraint location FAKE1 prefers test-centos-1=200
# pcs -f /tmp/FAKE_cfg constraint location FAKE1 avoids centos-vm1-cn
# pcs -f /tmp/FAKE_cfg constraint location FAKE1 avoids centos-vm2-cn
# pcs -f /tmp/FAKE_cfg constraint location FAKE2 prefers test-centos-2=200
# pcs -f /tmp/FAKE_cfg constraint location FAKE2 avoids centos-vm1-cn
# pcs -f /tmp/FAKE_cfg constraint location FAKE2 avoids centos-vm2-cn

Now we can push the FAKE Service configs to the cluster.

# pcs cluster cib-push /tmp/FAKE_cfg
CIB updated

Let's show the cluster status.

# pcs status
Cluster name: mycluster
Last updated: Tue Sep 15 15:20:33 2015		Last change: Tue Sep 15 15:20:23 2015 by root via cibadmin on centos-vm1-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
4 nodes and 10 resources configured

Online: [ centos-vm1-cn centos-vm2-cn ]
GuestOnline: [ test-centos-1@centos-vm2-cn test-centos-2@centos-vm2-cn ]

Full list of resources:

 ClusterIP	(ocf::heartbeat:IPaddr2):	Started centos-vm1-cn
 gluster-configs	(ocf::heartbeat:Filesystem):	Started centos-vm2-cn
 gluster-images	(ocf::heartbeat:Filesystem):	Started centos-vm2-cn
 libvirtd	(systemd:libvirtd):	Started centos-vm2-cn
 test-centos-1_vm	(ocf::heartbeat:VirtualDomain):	Started centos-vm2-cn
 test-centos-2_vm	(ocf::heartbeat:VirtualDomain):	Started centos-vm2-cn
 FAKE1	(ocf::pacemaker:Dummy):	Started test-centos-1
 FAKE2	(ocf::pacemaker:Dummy):	Started test-centos-2

PCSD Status:
  centos-vm1-cn: Online
  centos-vm2-cn: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Set Gluster Resource to Clone

One last thing we want to do is set GlusterFS resources to be cloned. This will start the resources on all hypervisor nodes, and will quicken the recovers after node failure.

To turn up GlusterFS Clone service resource, running the following commands on an offline configuration file.

# pcs cluster cib /tmp/GlusterFS-Clone_cfg

# pcs -f /tmp/GlusterFS-Clone_cfg resource clone gluster-configs
# pcs -f /tmp/GlusterFS-Clone_cfg resource clone gluster-images

Now we can push the GlusterFS Clone Service configs to the cluster.

# pcs cluster cib-push /tmp/GlusterFS-Clone_cfg
CIB updated

Let's show the cluster status.

# pcs status
Cluster name: mycluster
Last updated: Tue Sep 15 15:24:53 2015		Last change: Tue Sep 15 15:24:48 2015 by root via cibadmin on centos-vm1-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
4 nodes and 16 resources configured

Online: [ centos-vm1-cn centos-vm2-cn ]
GuestOnline: [ test-centos-1@centos-vm2-cn test-centos-2@centos-vm2-cn ]

Full list of resources:

 ClusterIP	(ocf::heartbeat:IPaddr2):	Started centos-vm1-cn
 libvirtd	(systemd:libvirtd):	Started centos-vm2-cn
 test-centos-1_vm	(ocf::heartbeat:VirtualDomain):	Started centos-vm2-cn
 test-centos-2_vm	(ocf::heartbeat:VirtualDomain):	Started centos-vm2-cn
 FAKE1	(ocf::pacemaker:Dummy):	Started test-centos-1
 FAKE2	(ocf::pacemaker:Dummy):	Started test-centos-2
 Clone Set: gluster-configs-clone [gluster-configs]
     Started: [ centos-vm1-cn centos-vm2-cn ]
     Stopped: [ test-centos-1 test-centos-2 ]
 Clone Set: gluster-images-clone [gluster-images]
     Started: [ centos-vm1-cn centos-vm2-cn ]
     Stopped: [ test-centos-1 test-centos-2 ]

PCSD Status:
  centos-vm1-cn: Online
  centos-vm2-cn: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Fencing

To configure fencing, there are several ways to do this, including isolating the network, isolating the power, or simply rebooting the machine. There are some that argue that the fence device should not share power with the device being fenced, since in the event of a power failure, the cluster will not be able to fence a failed device, and the cluster will remain in a partially failed state. This can be solved by configuring layered fencing, however this can be left to future discussions.

Configure Fencing and Constraints

The following commands will configure fencing on an offline file, which will be applied later. We set constraints so that the fencing resource on each node does not run on the node that it would act upon.

# pcs cluster cib /tmp/fencing_cfg

# pcs -f /tmp/fencing_cfg stonith create fence_centos-vm1_ipmi fence_ipmilan pcmk_host_list="centos-vm1-cn" ipaddr="centos-vm1-ipmi" login=fencer passwd=2eznRKdeTu7cEee op monitor interval=60s
# pcs -f /tmp/fencing_cfg constraint location fence_centos-vm1_ipmi avoids centos-vm1-cn

# pcs -f /tmp/fencing_cfg stonith create fence_centos-vm2_ipmi fence_ipmilan pcmk_host_list="centos-vm2-cn" ipaddr="centos-vm2-ipmi" login=fencer passwd=2eznRKdeTu7cEee op monitor interval=60s
# pcs -f /tmp/fencing_cfg constraint location fence_centos-vm2_ipmi avoids centos-vm2-cn

Now we can apply the config.

# pcs cluster cib-push /tmp/fencing_cfg

Finally, we can enable fencing on the cluster.

# pcs property set stonith-enabled=true

Fencing Delay

It may be useful to set a delay for the fencing devices. The default delay is 0, which means that the fence will act immediately upon failure. A delay would be useful to deal with momentart loss of communication between nodes, such as those caused by changes in switch topology.

If you wish to set a delay, you can run the following commands.

# pcs cluster cib /tmp/fencing-delay_cfg
# pcs -f /tmp/fencing-delay_cfg stonith update fence_centos-vm1_ipmi delay=15
# pcs -f /tmp/fencing-delay_cfg stonith update fence_centos-vm2_ipmi delay=15
# pcs cluster cib-push /tmp/fencing-delay_cfg

System Testing

For testing, it is useful to run the crm_mon command, which will continuously monitor the cluster state. Use <CTL>-C to exit the program.

Last updated: Wed Sep 16 16:17:14 2015          Last change: Wed Sep 16 16:11:54 2015 by root via cibadmin on centos-vm2-cn
Stack: corosync
Current DC: centos-vm2-cn (version 1.1.13-a14efad) - partition with quorum
4 nodes and 18 resources configured

Online: [ centos-vm1-cn centos-vm2-cn ]
GuestOnline: [ test-centos-1@centos-vm2-cn test-centos-2@centos-vm2-cn ]

ClusterIP	(ocf::heartbeat:IPaddr2):	Started centos-vm1-cn
libvirtd        (systemd:libvirtd):     Started centos-vm2-cn
test-centos-1_vm        (ocf::heartbeat:VirtualDomain): Started centos-vm2-cn
test-centos-2_vm        (ocf::heartbeat:VirtualDomain): Started centos-vm2-cn
FAKE1   (ocf::pacemaker:Dummy): Started test-centos-1
FAKE2   (ocf::pacemaker:Dummy): Started test-centos-2
 Clone Set: gluster-configs-clone [gluster-configs]
     Started: [ centos-vm1-cn centos-vm2-cn ]
 Clone Set: gluster-images-clone [gluster-images]
     Started: [ centos-vm1-cn centos-vm2-cn ]
fence_centos-vm1_ipmi   (stonith:fence_ipmilan):        Started centos-vm2-cn
fence_centos-vm2_ipmi   (stonith:fence_ipmilan):        Started centos-vm1-cn

Pacemaker_Remote Node Failure

To test a remote node failure, simply log into the node and run the following command.

# killall -9 pacemaker_remoted

You will see in crm-mon, where the remote node changed to FAILED state, and was immediately restarted.

Cluster Node Failure

To test a full cluster node, you will want to run the following command.

# killall -9 corosync

You will see in crm_mon where this node will go into "Offline/UNCLEAN" state. If you have fencing enabled without delay, it will reboot the node. If you have fencing disabled, or set for delay, you will see corosync restart, and the node rejoin the cluster.

You can do further testing by isolating the node, however care must be taken with a two-node cluster so that the isolation of the two nodes does not cause the nodes to reboot each other, resulting in a cluster without any nodes.

Shut Down PCS Cluster

You will be able to shut down a single node in a cluster using the following command. Shutting down the node gracefully will ensure that all resources are stopped, and that other nodes will restart those resources, as applicable, without enacting fencing and rebooting the node.

# pcs cluster stop

Other Observations While Testing

It has been observed that some resources will be restarted on the cluster under failure condition, even though those resources are not moving to another node. I have not been able to determine why this happens, only that it happens some of the time. Further research into this should be done.

Converting to Active/Active Cluster

To convert to an active/active cluster, we should only need to convert the libvirtd resource to a cloned resource, and then assign some location preference to the virtual machines.

Configure Libvirtd Clone

You can run the following commands to configure the libvirtd resource for cloning using an offline config file.

# pcs cluster cib /tmp/libvirtd-clone_cfg
# pcs -f /tmp/libvirtd-clone_cfg resource clone libvirtd

Now you can push the config to the cluster.

# pcs cluster cib-push /tmp/libvirtd-clone_cfg
CIB updated

Configure Resource Preference

You can determine where to normally run the resources by configuring location preferences.

We will do this on an offline config file.

# pcs cluster cib /tmp/vm-preferences_cfg
# pcs -f /tmp/vm-preferences_cfg constraint location test-centos-1_vm prefers centos-vm1-cn=200
# pcs -f /tmp/vm-preferences_cfg constraint location test-centos-1_vm prefers centos-vm2-cn=50
# pcs -f /tmp/vm-preferences_cfg constraint location test-centos-2_vm prefers centos-vm2-cn=200
# pcs -f /tmp/vm-preferences_cfg constraint location test-centos-2_vm prefers centos-vm1-cn=50

Now you can push the config to the cluster.

# pcs cluster cib-push /tmp/vm-preferences_cfg
CIB updated

Resource Stickiness

Stickness is the cost of moving a resource from it's present location. The default is unset (equivalent to 0). If the the location preference is higher than the stickiness value, then the resource will move to the preferred location, assuming it is available. If the location preference is lower than the stickiness value, then the resource will remain in the current location until further action is taken on the resource, or the location becomes unavailable due to maintenance or failure.

To set the default value, run the following command.

# pcs resource defaults resource-stickiness=100

To set the value per resource, teh commands are ran as shown below.

# pcs resource meta test-centos-1_vm resource-stickiness=250
# pcs resource meta test-centos-2_vm resource-stickiness=250

To unset stickiness, use one of the following forms.

# pcs resource defaults resource-stickiness=
# pcs resource meta test-centos-1_vm resource-stickiness=
# pcs resource meta test-centos-2_vm resource-stickiness=

Retry Start Failures

One of the frustrating issues that I have seen with Pacemaker is the fatality of failure to start. It appears that this can be partially solved with setting the start-failure-is-fatal default setting. By default, it is set to true so that any failure to start a resource will never be retried.

# pcs resource defaults start-failure-is-fatal=false
# pcs resource update <resource> meta failure-timeout="30s"
# pcs resource op delete <resource> start
# pcs resource op add <resource> start interval=0s timeout=90 on-fail="restart"
# pcs resource op delete <resource> monitor
# pcs resource op add <resource> monitor interval=10 timeout=30 on-fail="restart"

After set, the resource will look something like this:

# pcs resource show test-centos-1_vm
 Resource: test-centos-1_vm (class=ocf provider=heartbeat type=VirtualDomain)
  Attributes: hypervisor=qemu:///system config=/var/lib/libvirt/configs/test-centos-1.xml 
  Meta Attrs: remote-node=test-centos-1 resource-stickiness=250 failure-timeout=30s 
  Operations: stop interval=0s timeout=90 (test-centos-1_vm-stop-timeout-90)
              start interval=0s timeout=90 on-fail=restart (test-centos-1_vm-name-start-interval-0s-on-fail-restart-timeout-90)
              monitor interval=10 timeout=30 on-fail=restart (test-centos-1_vm-name-monitor-interval-10-on-fail-restart-timeout-30)

Manually Move Resource

You can manually move resources by running a series of commands.

# pcs constraint location test-centos-1_vm prefers centos-vm1-cn=INFINITY
# pcs constraint location test-centos-1_vm prefers centos-vm1-cn=200

First command sets the location preference to INFINITY, ensuring he move, if the location is available. The second command sets the location preference bact to what it was previously set to.

Adding Additional Nodes

You can add additional nodes using the below procedures, however you should always have an odd number of nodes to ensure that you are able to achieve quorum. Quorum is achieved when there are 50% of the configured nodes present, plus one additional node. The exception to this is the two-node cluster, where a single node will be able to achieve quorum.

Helpful Commands

crm_resource

Move Rsource

To move a resource manually from one node to another node, the following command will need to be set.

# crm_resource --resource  <resource> --move

OR

# crm_resource --resource <resource> --move --node <node>

NOTE: This will set the location preference on the current node to -INFINITY, which will need to be cleared to resume normal operations.

To permit, but not force a resource to return to normal operation, and return to preferred node on it's own.

# crm_resource --resource <resource> --un-move

pcs resource

Show Resources

This shows the defined resources, and their current configuration values.

# pcs resource show --full

pcs constraint

Show Constraints

To show constraints, use the following command:

# pcs constraint show

To show full constraints, including constraint ID, use the following:

# pcs constraint show --full

Additional Resources

Useful web links: