Kinetica installation and configuration instructions using KAgent for Microsoft Azure.
Prior to provisioning Azure instances for Kinetica, there are a few setup steps to prepare your Azure environment:
- Create a local SSH key pair
- Ensure the CPU quota for the desired region is at least 12; Kinetica runs best on instances with at least 12 CPU cores, so KAgent is limited to provisioning instances of that size or greater
- Register an application to an existing Azure Active Directory with the Contributor role assigned to it. Consult the Azure documentation for more information
- Create a client secret for the same application and store the value locally
- Ensure the supported regions & default security posture are commensurate with the targeted deployment.
Cluster Resiliency Considerations
Kinetica's cluster resiliency solution relies upon a shared file system accessible to all hosts in the cluster so that nodes and/or any potentially ailing processes can be failed over to a spare node (or other node) in the cluster. Spare nodes are configured in the Nodes configuration section of KAgent but can also be configured after installation. The shared file system must be in place prior to installing Kinetica with cluster resiliency enabled.
Using KAgent to provision to Microsoft Azure will setup a new Azure Resource Group and a shared file system with volumes being automatically provisioned to back the head node process and each worker node process in the cluster. If you already have an existing Azure Resource Group containing machines that are ready for Kinetica installation, you should use the on-premise installation instructions instead. Spare nodes provisioned after installation to an existing Azure setup will not have their volumes automatically provisioned; instead, the volume(s) will move with the head node and/or worker node process that is failing/switching over.
KAgent can be deployed as a RHEL or Debian/Ubuntu installation package on any server inside or outside the cluster. After copying the KAgent package to the target server, deploy it using the standard procedures for a local package:
sudo yum install ./kagent-<version>.<architecture>.rpm
sudo apt install ./kagent-<version>.<architecture>.deb
This installs the package to the directory
registers and starts the kagent_ui service. KAgent will open port 8081
on the local firewall (if enabled).
Installation of Kinetica on Azure using KAgent involves the automated deployment of the installation package via either a browser-based UI or console-driven CLI.
The installation process requires a license key. To receive a license key, contact support at firstname.lastname@example.org.
To access the KAgent UI and begin setting up a cluster:
Review KAgent for more information on KAgent and its features.
Ensure the KAgent service is started:
service kagent_ui status
Browse to the KAgent UI using IP or host name:
Optionally, if using custom rings, i.e. not the default ring, click Rings then click Clusters next to the desired ring. See High Availability Architecture for more information about rings and high availability (HA).
Click Add New or Existing Cluster.
Enter a name for the cluster. The name cannot contain spaces or underscores.
Optionally, select one or more of the following packages:
Select Core if node(s) in the cluster should have the core database functionality installed on them.
First-time setups should always have Core selected.
Select etcd if node(s) in the cluster should have etcd installed on them.
Select Graph if a node in the cluster should have the graph server installed on it. See Network Graphs & Solvers Concepts for more information.
Optionally, select to install AAW (Active Analytics Workbench) if a node should have AAW installed on it. See Active Analytics Workbench for more information.
Optionally, select to install KAgent if a node should also have KAgent installed on it. See KAgent for more information.
For the Install Mode, select either Online (install directly from the online Kinetica repository) or Offline (install from uploaded packages). If Offline is selected, click Upload Packages, then upload a package file for each component or driver desired for the installation.
If performing an offline installation, all necessary dependencies will need to be installed prior to cluster setup.
For the Version, select either CUDA (GPU) or Intel (CPU-only) to determine the package variant to install.
If the Version is set to CUDA, ensure Automatically install Nvidia driver is selected. This will automatically configure the server(s) for an Nvidia GPU driver and install the most compatible driver.
Enter the license key.
If AAW is selected to install, select a K8 Setup:
Automatic -- KAgent will install Kubernetes / KubeCTL and upload a default configuration file.
Before installing the embedded Kubernetes cluster, review the Kubernetes Considerations.
Custom -- Upload a configuration file for an already existing Kubernetes installation and provide a public IP address for the server hosting the Kubernetes installation. Note that AAW requires Kubernetes; see Active Analytics Workbench (AAW) Overview for more information.
KAgent does not support multiple installations of Kinetica with differing deployment types. For instance, a Microsoft Azure cluster and an AWS cluster cannot exist in the same instance of KAgent, even in different rings. Moreover, all Microsoft Azure clusters across all rings must exist in the same provider region.
- Select the Microsoft Azure deployment method.
- For Application (client) ID, provide the Client ID for the application mentioned in Prerequisites.
- For the Client Secret, provide the Client Secret for the application mentioned in Prerequisites.
- For the Subscription ID, provide the ID for the subscription that hosts the application mentioned in Prerequisites.
- For the Directory (tenant) ID, provide the Tenant ID for the application mentioned in Prerequisites.
- Select a Region .
- For SSH User, provide an SSH username that will be added to the provisioned instance(s).
- For SSH Public Key, optionally upload the SSH public key file to enable SSH connections to the instance(s).
- Click Next.
The Security configuration section is only required if Core is being installed.
Enter and confirm an Admin Password. It must meet the password strength requirements.
This is the password used to access Reveal, Active Analytics Workbench (AAW), KAgent, and GAdmin as the default Admin user.
Select an SSL Mode:
- Cert/key setup not required -- Kinetica will not require SSL certificate/key creation/upload and SSL will not be enabled
- User-provided cert/key per node -- user must upload an SSL
certificate and key for each node; Kinetica copies the cert/key pair
/opt/gpudb/certs, enables HTTPD, and configures HTTPD to use HTTPS
- Generate self-signed cert/key per node -- KAgent generates
a self-signed certificate and key for each node and places it in
/opt/gpudb/certs, enables HTTPD, and configures HTTPD to use HTTPS
Select an Authentication type and fill the fields as necessary:
- None -- no authentication or authorization
- LDAP -- configures Kinetica to authenticate via LDAP; requires authentication to connect to the database, enables authorization, enables external authentication, automatically creates users in the database for LDAP users, and automatically grants roles in the database to LDAP users
- Active Directory -- configures Kinetica to authenticate via Microsoft Active Directory; requires authentication to connect to the database, enables authorization, enables external authentication, automatically creates users in the database for Active Directory users, and automatically grants roles in the database to Active Directory users
- Kerberos -- configures Kinetica to authenticate via Kerberos; requires authentication to connect to the database, enables authorization, enables external authentication, automatically creates users in the database for Kerberos users, and automatically grants roles in the database to Kerberos users
No SSL or authentication is not recommended! For more information on security configurations and settings as well as how to manually configure Kinetica for a secure setup, see Security Configuration.
Click Add New Node until there are the desired number of nodes that will have Kinetica (and potentially other services) installed on them.
If the User-provided cert/key per node SSL Mode was selected in Security, an SSL column will be added to the configuration page--click the lock icon in the SSL column to open the SSL Certificate/Key window, where the SSL cert and key, along with an optional public hostname, can be provided. Repeat this for each node.
Optionally, select if each node should have the Core package installed. The Core package contains access to the database and its core components and functionality. Note that if the core package is not installed on a node, that node cannot be designated as the Head Node.
Optionally, select if each node should have the Core package installed. The Core package contains access to the database and its core components and functionality. Note that if the core package is installed on a node, that node cannot be designated as the Head Node.
Optionally, select if each node should have the etcd package installed. The etcd package provides a means for each node in the cluster to have a consistent record of statuses and locations for the other node(s) in the cluster. Ensure at least one node will have etcd installed; select additional nodes for redundancy.
Select the desired node for the Head Node using the corresponding radio button. This server will receive user requests and parcel them out to the other worker nodes of the system. The head node of the cluster (or only node in a single-node system) will also be used for the administration of the cluster, and by default, the hosting of Reveal and GAdmin and as such, will require special handling during the installation process.
All services and privileges (Head, Graph, AAW, etc.) can exist on a single node if desired, assuming there are enough resources to handle it.
If the Graph package was selected for install in Cluster, select the desired node(s) to host the graph service using the corresponding radio button. The graph node does not need to have the Core package enabled. Consult Distributed Graph Servers for more information on leveraging multiple graph servers.
If the AAW package was selected for install in Cluster, optionally set the number of reserved GPUs for AAW to use for each node. The AAW service will co-exist with the head node. Note that the AAW package will be automatically installed on every node with the Core package enabled. If enabling cluster resiliency, consult the head node failover section for information on how cluster resiliency works with AAW.
Some features of AAW require GPUs to work or have increased performance. Review the AAW documentation for more information.
If the KAgent package was selected for install in Cluster, select the desired node to host the service. The KAgent node does not need to have the Core package enabled.
If the RabbitMQ package was selected for install in Cluster because a High Availability setup is required, select the desired node(s) to have RabbitMQ installed. Ensure at least one node will have RabbitMQ installed if enabling High Availability (HA) for the cluster; select additional nodes to have RabbitMQ installed for redundant queues, but note that in total, an odd number of nodes should be selected for RabbitMQ installation. A node does not have to host any other services other than RabbitMQ if desired.
In total, an odd number of nodes should be selected for RabbitMQ installation. Kinetica recommends installing RabbitMQ machines that will not have the Core package enabled to achieve better resiliency especially in cases where node failover is enabled.
Optionally, select Spare if a node should be a spare. A spare node cannot be the initial host of any other component.
If a node was marked as a spare, confirm that the system already has the required distributed/shared file system installed, then click OK.
- For the Server SSH Credentials, upload the SSH private key that will be used to access the node(s). This key should match the public key uploaded in Deployment.
- Click Next.
Review the Installation Summary to ensure there are no validation errors in the information.
Click CLI Commands to view and/or copy the KAgent command line interface commands that will be run in the background (order is from top to bottom).
Click Install. KAgent will open a window displaying the progress of the installation.
Click Details next to a step to see stdout and stderr for that step. Click to copy the displayed text.
The installation may take a while as KAgent initializes each node in the
cluster, verifies the cluster, adds a repository, downloads the package, installs
the package to the directory
/opt/gpudb, creates a group named
gpudb, and creates two users (gpudb & gpudb_proc) whose home
directories are located in
/home/gpudb. This will also register two
services: gpudb & gpudb_host_manager.
If Automatic Kubernetes (K8) installation was selected and Kinetica is being installed on a RHEL-based system, KAgent will request permission to disable SELinux on the nodes. Kubernetes cannot be installed otherwise. Click I Agree to continue with the installation; click No to stop the installation and manually disable SELinux.
After a successful installation, if KAgent was also installed on a separate node, one can be redirected to the KAgent on that cluster node. If KAgent was not installed on a separate node, one can be redirected to Kinetica Administration Application (GAdmin).
After the installation, the cluster will be added to KAgent and you'll be logged into KAgent as the admin user for the cluster. After this session is over (via either logging out or session timeout), you'll be required to log into KAgent every time you want to access KAgent features. See Logging In / Out for more information.
To validate that Kinetica has been installed and started properly, you can perform the following tests.
To ensure that Kinetica has started (you may have to wait a moment while the system initializes), you can run curl on the head node to check if the server is responding and port is available with respect to any running firewalls:
$ curl localhost:9191 Kinetica is running!
You can also run a test to ensure that the API is responding properly. There is an admin simulator project in Python provided with the Python API, which pulls statistics from the Kinetica instance. Running this on the head node, passing in the appropriate <username> & <password>, you should see:
$ /opt/gpudb/bin/gpudb_python /opt/gpudb/kitools/gadmin_sim.py -u <username> -p <password> --table --summary +-----------------+--------------------------------+----------------------+----------------------+-------+ | Schema | Table/View | Records | Type ID | TTL | +=================+================================+======================+======================+=======+ | SYSTEM | <ALL TABLES/VIEWS> | 1 | | | | SYSTEM | ITER | 1 | UNSET_TYPE_ID | -1 | +-----------------+--------------------------------+----------------------+----------------------+-------+ +---------------------------+----------------------+ | Object Type | Count | +===========================+======================+ | Schemas | 1 | | Tables & Views | 1 | | Records | 1 | | Records + Track Elements | 1 | +---------------------------+----------------------+
GAdmin Status Test
The administrative interface itself can be used to validate that the system is functioning properly. Simply log into GAdmin. Browse to Dashboard to view the status of the overall system and Ranks to view the status breakdown by rank.
After verifying Kinetica has started and its components work, you should confirm ingesting and reading data works as expected.
- Navigate to the Demo tab on the Cluster page.
- Click Load Sample Data under the NYC Taxi section, then click Load to confirm.
- Once the data is finished loading, click View Loaded Data. The data should be available in the nyctaxi table located in the demo schema.
If Reveal is enabled:
Log into Reveal and change the administration account's default password.
Click NYC Taxi under Dashboards. The default NYC Taxi dashboard should load.
Kinetica comes packaged with many helpful server and support executables that can be found in /opt/gpudb/core/bin/ and /opt/gpudb/bin. Note that any of the gpudb_hosts_*.sh scripts will operate on the hosts specified in gpudb.conf. Run any of the following with the -h option for usage information.
For most of the utilities that use passwordless SSH, an AWS PEM file can be specified instead using the -i option (with the exception being the gpudb_hosts_persist_* scripts). If passwordless SSH is not setup and no PEM file is specified, you will be prompted for a password on each host.
Environment Configuration and Tools
Some of the most commonly used and important utilities are also available in the /opt/gpudb/bin directory.
This directory also contains the KI Tools suite
|Utility / Script||Uses Passwordless SSH||Description|
|gpudb_alter_password||No||Script to change a given user's password|
|gpudb_env||No||Utility to run a program and its given arguments after setting the PATH, LD_LIBRARY_PATH, PYTHON_PATH, and others to the appropriate /opt/gpudb/ directories. Use this script or /opt/gpudb/bin/gpudb_python to correctly setup the environment to run Kinetica's packaged Python version. You can also run source /opt/gpudb/core/bin/gpudb_env.sh to have the current environment updated.|
|gpudb_pip||Yes||Script to run Kinetica's packaged pip version. Runs on all hosts. This can be used in place of pip, e.g., /opt/gpudb/bin/gpudb_pip install gpudb|
|gpudb_python||No||Script to correctly setup the environment to run Kinetica's packaged Python version. This can be used in place of the python command, e.g., /opt/gpubd/bin/gpudb_python my_python_file.py|
|gpudb_udf_distribute_thirdparty||No||Utility to mirror the local /opt/gpudb/udf/thirdparty to remote hosts. Creates a dated backup on the remote host before copying|
Additional helper scripts and utilities are available in /opt/gpudb/core/bin.
|Utility / Script||Uses Passwordless SSH||Description|
|gpudb||No||Run as gpudb user or root. The Kinetica system start/restart/stop/status script|
|gpudb_alter_password.py||No||Script to change a given user's password|
|gpudb_cluster_cuda||No||Server executable for CUDA clusters. Displays version and configuration information. This should only be run by the gpudb executable (see above).|
|gpudb_cluster_intel||No||Server executable for Intel clusters. Displays version and configuration information. This should only be run by the gpudb executable (see above).|
|gpudb_conf_parser.py||No||Run using /opt/gpudb/bin/gpudb_python. Utility for parsing the /opt/gpudb/core/etc/gpudb.conf file and printing the settings and values.|
|gpudb_config_compare.py||No||Script to compare two configuration files: a "modified" configuration file and a "baseline" configuration file. The script can also merge the files after outputting the diff. The merged file will use the "modified" file's settings values if the "modified" configuration settings match the "baseline" configuration settings; if a setting value is present in the "modified" file but not in the "baseline" file, the "baseline" setting value will be used. Supports .ini, .conf, .config, .py, and .json files.|
|gpudb_decrypt.sh||No||Utility for decrypting text encrypted by gpudb_encrypt.sh. See Obfuscating Plain-Text Passwords for details.|
|gpudb_disk_mount_azure.sh||No||Utility used for attaching and detaching data volumes for Kinetica clusters running in Microsoft Azure.|
|gpudb_encrypt.sh||No||Utility for encrypting text. See Obfuscating Plain-Text Passwords for details.|
|gpudb_env.sh||No||Utility to run a program and its given arguments after setting the PATH, LD_LIBRARY_PATH, PYTHON_PATH, and others to the appropriate /opt/gpudb/ directories. Use this script or /opt/gpudb/bin/gpudb_python to correctly setup the environment to setup the environment to run Kinetica's packaged Python version. You can also run source /opt/gpudb/core/bin/gpudb_env.sh to have the current environment updated.|
|gpudb_file_integrity_check.py||No||Utility to test the consistency of the /opt/gpudb/persist directory|
|gpudb_generate_key.sh||No||Utility for generating an encryption key. See Obfuscating Plain-Text Passwords for details.|
|gpudb_host_manager||No||The host daemon process that starts and manages any Kinetica processes.|
|gpudb_hosts_addresses.sh||Yes||Prints all the unique hostnames (or IPs) specified in gpudb.conf|
|gpudb_hosts_diff_file.sh||Yes||Run as gpudb user or root. Utility to diff a given file from the current machine to the specified destination file on one or more hosts|
|gpudb_hosts_logfile_cleanup.sh||Yes||Run as gpudb user or root. Script to delete old log files and optionally keep the last n logs|
Run as gpudb user or root. Script to clear the database persist files (location specified in gpudb.conf)
Important: Only run this while the database is stopped.
|gpudb_hosts_rsync_to.sh||Yes||Run as gpudb user. Script to copy files from this server to the remove servers using rsync|
Run as gpudb user or root. Script to distribute the gpudb user's public SSH keys to the other hosts defined in gpudb.conf to allow password-less SSH. This script should only be run from the head node.
Important: This script should be re-run after changing the host configuration to redistribute the keys
|gpudb_hosts_ssh_execute.sh||Yes||Run as gpudb user or root. Script to execute a program with arguments on all hosts specified in gpudb.conf, e.g., ./gpudb_hosts_ssh_execute.sh "ps aux" or ./gpudb_hosts_ssh_execute.sh "hostname"|
|gpudb_hosts_ssh_setup_passwordless.sh||Yes||Script to add an authorized SSH key for a given user across a set of hosts.|
|gpudb_keygen||No||Executable to generate and print a machine key. You can use the key to obtain a license from email@example.com|
|gpudb_log_plot_job_completed_time.sh||No||Plots job completion time statistics using gnuplot|
|gpudb_machine_info.sh||No||Script to print OS config information that affects performance as well as suggestions to improve performance|
|gpudb_migrate_persistence.py||No||Utility to migrate data from a local persist directory into the database|
|gpudb_nvidia_setup.sh||No||Utility to configure the Nvidia GPU devices for best performance or restore defaults. Root permission is required to change values. Utility reports informational settings and permission errors when run as user|
|gpudb_open_files.sh||No||Script to print the files currently open by the database|
|gpudb_process_monitor.py||No||Script to check a process list against a matching regular expression and print a log to stdout when the process is started or stopped. The script can also run a program, send emails, and/or SNMP alerts when the process starts or stops. The script can be configured using a configuration file, but note that some settings can be overridden from the command line.|
|gpudb_sysinfo.sh||No||More information when run as root. Script to print a variety of information about the system and hardware for debugging. You can also make a .tgz file of the output. Rerun this program as needed to keep records of the system. Use a visual diff program to compare two or more system catalogs|
|gpudb_udf_distribute_thirdparty.sh||Yes||Utility to mirror the local /opt/gpudb/udf/thirdparty to remote hosts. Creates a dated backup on the remote host before copying|
|gpudb_useradd.sh||No||Script to create the gpudb:gpudb and gpudb_proc:gpudb_proc user:groups and SSH id. This script can be rerun as needed to restore the user:groups and ssh config. Be sure to rerun (on the head node only) gpudb_hosts_ssh_copy_id.sh to redistribute the SSH keys if desired whenever the SSH keys are changed|
The best way to troubleshoot any issues is by searching through the available logs. For more information on changing the format of the logs, see Custom Logging. Each component in Kinetica has its own log, the location of which is detailed below:
|Active Analytics Workbench (AAW) (API)|
|Active Analytics Workbench (AAW) (UI)|
|Kinetica system logs|
If additional edits to the database's configuration file are desired, e.g.,
UDFs (procs), auditing, etc., the database will need to be stopped and the
file will need to be updated. System configuration is done primarily through the
/opt/gpudb/core/etc/gpudb.conf, and while all nodes
in a cluster have this file, only the copy on the head node needs to be
modified. The configuration file can be edited via GAdmin or via a text editor
on the command line.
Only edit the
/opt/gpudb/core/etc/gpudb.conf on the
head node. Editing the file on worker nodes is not supported and may
lead to unexpected results.
Some common configuration options to consider updating:
Enabling multi-head ingest
Changing the persist directory
The directory should meet the following criteria:
- Available disk space that is at least 4x memory
- Writable by the gpudb user
- Consist of raided SSDs
- Not be part of a network share or NFS mount
Enabling UDFs (procs)
Adjusting storage tiers and resource groups
To edit the configuration file via GAdmin:
- Log into GAdmin
- Enter admin for the Username
- Enter the Admin Password provided to KAgent for the Password (refer to KAgent UI for more information)
- Click Log In
- Stop the system.
- Navigate to
- Edit the file in the text window.
- Click Update, then click Start Service.
To edit the configuration file via command line: