High Availability Installation & Configuration

Pre-requisites

An HA installation requires three components:

1. Two or more clusters with matching Kinetica installations
2. A Kinetica HA installation package that matches the cluster installation
3. A load balancer fronting all clusters in sticky IP/IP hashing mode

Note

While HA can operate in an environment without a load balancer, it will function in a failover-only mode; to make full use of all clusters simultaneously, a load balancer is necessary.

Install

Install the HA Plugin RPM or DEB on the head node of each Kinetica cluster. This will create the directory /opt/gpudb/ha.

• On RHEL:

  yum -y install gpudb-ha-<version>-<release>.x86_64.rpm
  

• On Debian:

  apt-get install -f gpudb-ha-<version>-<release>.x86_64.deb
  

Configure Core

1. Shut down all clusters:

   sudo service gpudb stop
   

2. Edit the /opt/gpudb/core/etc/gpudb.conf file on the head node of each cluster for the following:

   1. Update the head node port to 9391:

      sed -i 's/^head_port.*/head_port = 9391/' /opt/gpudb/core/etc/gpudb.conf
      

   2. Enable HA:

      enable_ha = true
      

   3. Optionally, set the ha_use_https setting to true if HA will will be configured to use HTTPS:

      ha_use_https = true
      

Configure HA

The Kinetica HA configuration parameters are in the /opt/gpudb/ha/conf/ha-proc-conf.xml XML file. In a single head node environment, these files will only need to be updated on the head node; in a multiple head node environment, these files will need to be updated on every node in every cluster. In this file, child refers to the head node of the cluster being configured, and brother refers to the head nodes of the other clusters. Below are the parameters you will have to change along with sample values for a three cluster HA configuration, with head nodes of each cluster named ha1, ha2, and ha3.

Parameter	Purpose	ha1	ha2	ha3
child.store	URL of the head node	http://127.0.0.1:9391	http://127.0.0.1:9391	http://127.0.0.1:9391
child.store.name	Domain name of cluster's head node	ha1	ha2	ha3
brother.stores	URLs of the other clusters	http://ha2:9391,http://ha3:9391	http://ha1:9391,http://ha3:9391	http://ha1:9391,http://ha2:9391
msgq.url	Message queues domain names	ha1,ha2,ha3	ha1,ha2,ha3	ha1,ha2,ha3
main.exchange.name	Exchanges used to replicate data to brother/backup queues	gpudb_exchg	gpudb_exchg	gpudb_exchg
child.queue.name	Child queue names; must start with gpudb_ to ensure replication across hosts	gpudb_ha1	gpudb_ha2	gpudb_ha3
brother.queue.names	Brother queue names; must start with gpudb_ to ensure replication across hosts	gpudb_ha2,gpudbha3	gpudb_ha1,gpudb_ha3	gpudb_ha1,gpudb_ha2
rabbitmq.user	RabbitMQ username	gpudb	gpudb	gpudb
rabbitmq.password	RabbitMQ password	<password>	<password>	<password>

Important

Ensure the message queue domain name listings for the msgq.url setting use short hostnames instead of long hostnames, e.g., ip-172-123-45-67 instead of ip-172-123-45-67.ec2.internal

Configure RabbitMQ

Kinetica HA uses RabbitMQ to sync data and commands between clusters. The configuration for these queues is on the head node of each cluster, at /opt/gpudb/ha/rabbitmq-server/conf. Below is an example of one of these files. The first file you will need to change is rabbitmq.config.

Important

If using multi-head ingest or read, this configuration is only necessary for the RabbitMQ nodes.

[
        {rabbit,
                [
                        {default_user,        <<"gpudb">>},
                        {default_pass,        <<"gpudb123">>},
                        {cluster_nodes,
                                {[
                                'rabbit@ha1','rabbit@ha2','rabbit@ha3'
                                ], disc}
                        },
                        {loopback_users, []}
%%                      ,{collect_statistics, fine}
%%                      ,{cluster_partition_handling, pause_minority}
%%                      ,{delegate_count, 64}
%%                      ,{hipe_compile, true}
                ]
        }
].

If you are not using the server hostnames as the server name in your queues, you will need to edit the file rabbitmq-env.conf on each cluster head node, and uncomment and change the line

NODENAME=rabbit@<server name>

Start RabbitMQ on each cluster's head node:

service gpudb-ha mq-start

From a browser on each head node, log into that node's respective RabbitMQ page at http://ha1:15672/#/, http://ha2:15672/#/, or http://ha3:15672/#/ and ensure the head node can access the RabbitMQ instance and message queue associated with it. The login information is configured in rabbitmq.config above.

Configure GAdmin

You can centrally manage all Kinetica clusters in an HA configuration through the Kinetica administration application (GAdmin) by editing its configuration files. This application can be run from any of the clusters or on a separate server. Unlike the HA configuration described above, this only needs to be done on one server.

1. In the file /opt/gpudb/tomcat/webapps/gadmin/js/settings.js, change the HA_ENABLED line to true

   var HA_ENABLED = true;
   

2. Start Kinetica HA:

   service gpudb start
   service gpudb-ha start
   

3. Copy the file ha-config.json (which gets created when gpudb-ha is started) to ha-sys-config.json. If you are running GAdmin from a separate server, you will need to copy this file from the /opt/gpudb/tomcat/webapps/gadmin directory on one of the cluster head nodes.

4. Log into GAdmin on the centrally-managed head node (or each cluster's head node, if not using central management), and perform & verify the following actions though it:

   • Create a table
   • Insert data into the table
   • Read data from the table
   • Delete the table
   • Create a user
   • Delete the user

Enabling HA Startup Scripts

The HA service should be added to chkconfig so it will be started during system reboot. Following the instructions above should result in the necessary scripts being available in /etc/init.d. To enable HA startup scripts:

chkconfig --add gpudb-ha

Managing HA

See Managing an HA Cluster for more information.