Alerting

Kinetica provides a simple alerting system controlled by Host Manager that can monitor the health of the system and resource usage.

Alerts are enabled by default and pre-configured to report on errors, critical system states, and a nominal set of resource usage thresholds.

If an alert is triggered, the alert is stored in memory and also logged to alert_store.json in the head node's persist directory. In a default configuration that file is here:

/opt/gpudb/persist/gpudb/rank-0/alert_store.json

The alert is viewable through the Kinetica Administration Application under Alerts. The /admin/show/alerts endpoint can be used to retrieve previous alerts.

If specified, the system can run a user-provided executable upon receiving each alert. See Run an Executable for details.

Alert Parameters

Below is a table containing the different alert types and the parameters associated with each.

Alert Type	Parameters
host_status	host [host IP] status [status code]
rank_status	rank [rank number] status [status code]
memory_absolute	host [host IP] threshold [threshold crossed] available_memory [free mem in bytes] total_memory [total mem on host] percentage_space [percent free mem]
memory_percentage	host [host IP] threshold [threshold crossed] available_memory [free mem in bytes] total_memory [total mem on host] percentage_space [percent free mem]
disk_absolute	directory [directory where threshold crossed] host [host IP] threshold [threshold crossed] available_space [free space in bytes] total_space [total space on disk] percentage_space [percentage of free disk space]
disk_percentage	directory [directory where threshold crossed] host [host IP] threshold [threshold crossed] available_space [free space in bytes] total_space [total space on disk] percentage_space [percentage of free disk space]
rank_cuda_error	rank [rank where error occurred] error_count [number of calls with errors] error_code [most recent cuda error code] error_string [most recent error message]
rank_fallback_allocator	failure_count [number of GPU memory allocation failures] min_failure_size [smallest allocation that failed in bytes] avg_failure_size [average size of allocation failure] max_failure_size [largest allocation that failed in bytes]
error_message	rank [rank where error occurred] topic [1-2 word string that indicates what functionality caused the error] message [free text description of error wrapped in single quotes]

Using Alerts

To enable alerts, navigate to /opt/gpudb/core/etc/gpudb.conf and modify the enable_alerts parameter:

enable_alerts = true

Alerting can be modified in any of the following ways by setting the appropriate parameter in /opt/gpudb/core/etc/gpudb.conf:

• Run an Executable
• Status Changes
• CUDA Errors
• Fallback Allocator Errors
• System Errors
• Memory Usage
• Disk Usage
• Maximum Alerts

Run an Executable

An executable can be run if an alert is triggered. The executable cannot have command-line parameters as the system passes parameters that pertain to the alert itself. The executable must be present on the host where the head node is located and it must be executable by the gpudb user. The executable does not need to be present on other hosts.

alert_exe = /file/path/to/script

The executable will be passed alert strings in the form:

[alert_type] --[parameter1] [value1] ... --[parameterN] [valueN]

For instance:

rank_status --rank 0 --status running

Important

The order of command line parameters passed to the executable can't be guaranteed.

Status Changes

Alerts can be generated if the status of a host or rank changes. Filters can be applied which govern which statuses generate alerts. All the statuses are available here.

alert_host_status = true
alert_host_status_filter = fatal_init_error
alert_rank_status = true
alert_rank_status_filter = fatal_init_error, not_responding, terminated

Important

The host and rank status filters are inclusive, alerting only on the specified statuses; e.g., alert_host_status_filter = fatal_init_error, shutdown would only display alerts for fatal init errors and shutdown events. Leave filter setting blank to alert on all status changes.

CUDA Errors

CUDA errors on ranks can generate alerts.

alert_rank_cuda_error = true

Fallback Allocator Errors

Rank fallback allocator errors can generate alerts.

alert_rank_fallback_allocator = true

System Errors

System errors can generate alerts in the event of significant or noteworthy runtime issues.

alert_error_messages = true

Memory Usage

Memory usage alerts can be generated based on either an absolute threshold (in bytes) or a percentage. These thresholds will be measured against the available host memory as reported by sysconf.

alert_memory_absolute = <mem-free-in-bytes>

# or

alert_memory_percentage = <mem-free-percent>

Note

While only one type of threshold can be enabled, the settings support multiple values, e.g., alert_memory_percent = 20, 10, 5, 1. However, if multiple thresholds are crossed, only the lowest one will trigger an alert.

Disk Usage

Disk usage alerts can be generated based on either an absolute threshold (in bytes) or a percentage. These thresholds will be measured against the available disk space on the drive(s) hosting each persist directory.

alert_disk_absolute = <disk-free-in-bytes>

# or

alert_disk_percentage = <disk-free-percent>

The monitored persist directories are those specified in the following /opt/gpudb/core/etc/gpudb.conf settings:

• persist_directory
• data_directory
• object_directory
• sms_directory
• text_index_directory

Note

While only one type of threshold can be enabled, the settings support multiple values, e.g., alert_disk_percent = 20, 10, 5, 1. However, if multiple thresholds are crossed, only the lowest one will trigger an alert.

Maximum Alerts

The number of generated alerts stored in memory/disk can be limited to a specified maximum.

alert_max_stored_alerts = 100