In this blog post, we will talk a little bit about High Availability and a little bit more about Pacemaker. Here are some of the topics of this post:
Introduction to High Availability (HA) and Clustering.
Benefits of Highly Available applications.
How HA is implemented on Red Hat Enterprise Linux (RHEL) 7.
HA requirements on RHEL 7.
Demo: Building a 3 node Apache cluster.
Introduction to High Availability (HA) and Clustering.
What is High Availability (HA)?
In IT, High Availability refers to a system or component that is continuously operational for a desirably long length of time.
3 cores principles to HA
Elimination of single point of failures.
Detection of failures as they occur.
How High Availability is implemented on RHEL 7?
What is Clustering?
A cluster is a set of computers working together on a single task. Which task is performed, and how that task is performed, differs from cluster to cluster.
There are four (4) different kinds of clusters:
High-availability clusters: Known as an HA cluster or failover cluster, their function is to keep running services as available as they can be. You could find them in two main configurations:
- Active-Active (where a service runs on multiple nodes).
- Active-Passive (where a service only runs on one node at a time).
Load-balancing clusters: All nodes run the same software and perform the same task at the same time and the requests from the clients are distributed between all the nodes.
Compute clusters: Also know as high-performance computing (HPC) cluster. In these clusters, tasks are divided into smaller chunks, which then get computed on different nodes.
Storage clusters: All nodes provide a single cluster file system that will be used by clients to read and write data simultaneously.
Benefits of Highly Available applications
In two words: "Application resiliency".
- Apply patches.
- Planned outages.
- Unplanned outages due to failures (server, software, network, storage).
How HA is implemented on RHEL 7?
Red Hat Enterprise Linux High Availability Add-On.
The High Availability Add-On consists of the following major components:
Cluster infrastructure: Provides fundamental functions for nodes to work together as a cluster: configuration file management, membership management, lock management, and fencing.
High-availability Service Management: Provides failover of services from one cluster node to another.
Cluster administration tools: Configuration and management tools for setting up, configuring, and management.
To provide the above services multiple
software components are required on the cluster nodes.
The cluster infrastructure software is provided by Pacemaker and performs the next set of functions:
- Cluster management
- Lock management
- Cluster configuration management
pacemaker: It's responsible for all cluster-related activities, such as monitoring cluster membership, managing the services and resources, and fencing cluster members. The RPM contains three (3) important components:
- Cluster Information Base (CIB).
- Cluster Resource Management Daemon (CRMd).
corosync: This is the framework used by Pacemaker for handling communication between the cluster nodes.
pcs: Provides a command-line interface to create, configure, and control every aspect of a Pacemaker/corosync cluster.
Requirements and Support.
Here are some requirements and limits for Pacemaker.
Number of Nodes:
- Up to 16 nodes per cluster.
- Minimum number of nodes: 3.
- 2 nodes cluster could be configured but is not recommended.
Single site: A cluster setup where all cluster members are in the same physical location, connected by a local area network. (Supported).
Multisite: Two clusters, one active and one for disaster recovery. Failover for multisite clusters must be managed manually. (Supported).
Strech (or) Geo Clusters: Clusters stretched out over multiple physical locations. (Required architecture review to be supported).
Fencing is the process of cutting a node off from shared storage. This can be done by power cycling a node or disabling communication to the storage level.
WARNING: Fencing is required for all nodes in the cluster, either via power fencing, storage fencing, or a combination of both.
NOTE: If the cluster will use integrated fencing devices like ILO or DRAC, the systems acting as cluster nodes must power off immediately when a shutdown signal is received, instead of initiating a clean shutdown.
Virtual Machines supported as nodes and resources.
NOTE: VM as a resource means that virtualization host is participating in a cluster and the virtual machine is a resource that can move between cluster nodes.
- Multicast and IGMP (Internet Group Management Protocol).
- Gratuitous ARP used for floating IP Address.
- 5405/UDP - corosync
- 2224/TCP - pcsd
- 3121/TCP - pacemaker
- 21064/TCP - dlm
Building a 3 node Apache cluster.
Now, we will build a basic pacemaker cluster serving Apache. To do this we will need 3 systems running RHEL 7.X.
Preparing the systems :
All these actions will happen on the all the cluster nodes.
Let's start by configuring FirewallD to allow traffic.
firewall-cmd --permanent --add-service=high-availability firewall-cmd --reload
Install required software
yum install pcs fence-agents-all
The pcs package requires corosync and pacemaker, so all your software be installed by doing this._
pcsd provides cluster configuration sync and the web front end. Needs to be enable in all the servers.
systemctl enable pcsd; systemctl start pcsd
Set the ha cluster user password
After the software install, a user hacluster will be created. This user will be used for all cluster communication pcsd.
NOTE: You should use the same password across all cluster nodes for this user. If you echo your password like is show below, clear your history afterward :) .
echo password | passwd hacluster --stdin
You should be able to resolve all the nodes in the cluster by name. On this example, we are going to use host files to define our 3 nodes. This is what I added to my hosts files (/etc/hosts).
192.168.1.10 node1 node1.example.local 192.168.1.20 node2 node2.example.local 192.168.1.30 node3 node3.example.local
Preparing the systems
pcsd requires that the cluster nodes authenticate, we are going to use the ha cluster user and password. This actions only needs to happen on one of the nodes.
[[email protected]] pcs cluster auth node1.example.local node2.example.local node3.example.local
Creating the cluster
Let's create the cluster:
pcs cluster setup --name demo-cluster --start node1.example.local node2.example.local node3.example.local
An important step will be to enable the cluster services on all nodes. By default, if a node is rebooted will not join the cluster until started manually. To avoid this do:
[[email protected]] pcs cluster enable --all
Check the cluster status:
[[email protected]] pcs cluster status
This is a critical step, you must have fencing on the cluster! In this example, I am assuming that we are using KVM for the demo and we will use fence_xvm.
[[email protected]] pcs stonith create fence_node1_vm fence_xvm port="node1" pcmk_host_list="node1.example.local" [[email protected]] pcs stonith create fence_node2_vm fence_xvm port="node2" pcmk_host_list="node2.example.local" [[email protected]] pcs stonith create fence_node3_vm fence_xvm port="node3" pcmk_host_list="node3.example.local"
Open the port for the fencing agent fencing:
[[email protected]] for i in `seq 1 3`; do ssh [email protected]$i.example.local firewall-cmd --add-port=1229/tcp --permanent; done [[email protected]] for i in `seq 1 3`; do ssh [email protected]$i.example.local firewall-cmd reload; done
Check fencing status.
[[email protected]] pcs stonith show
Clustered services consist of one or more resources. A resource can be:
- IP address
- file system
- Service (example: httpd)
Also usually the resources are a member of resource groups.
Creating the resources for our demo-cluster
First let's create the resource group for our Apache cluster, we are going to name it personal-web and will have a floating IP.
[[email protected]] pcs resource create floatingip IPaddr2 ip=192.168.1.254 cidr_netmask=24 --group personal-web
Install Apache (httpd)
[[email protected]] yum install httpd -y
Create the web-1 resource using Apache and put it on the personal-web group.
[ro[email protected]] pcs resource create web-1 apache --group personal-web
Let's check that the created resources are present in the cluster configuration and check the cluster status.
[[email protected]] pcs resource show [[email protected]] pcs status
Create a file in /var/www/html/ with this content:
echo "Website responding from $HOSTNAME" > /var/www/html/index.html
Manipulating the cluster
Here are some commands that will help you to manage the cluster.
Start and Stop the cluster
to do it in all nodes use the --all switch.
[[email protected]] pcs cluster start [[email protected]] pcs cluster stop
Stop the cluster service on a specific remote node
[[email protected]] pcs cluster stop node2.example.com
Disable cluster on reboot on a node
[[email protected]] pcs cluster disable
How to add a node to the cluster
[[email protected]] pcs cluster node add new.example.com
On the new node, you need to Authenticate the rest of the cluster. Also, you will need to add a fence device for it too.
[[email protected]] pcs cluster auth
How to remove a node to the cluster
[[email protected]] pcs cluster node remove new.example.com [[email protected]] pcs stonith remove fence_newnode new.example.com
Set a node in standby
(This bans the node from hosting resources).
[[email protected]] pcs cluster node standby new.example.com
Set the cluster in standby
[[email protected]] pcs cluster standby --all
[[email protected]] pcs cluster unstandby --all
Displaying quorum status
[[email protected]] corosync-quorumtool
High Availability Add-On overview: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Overview/
Cluster Labs website: