Resource Management Configuration

Global Tier Parameters

The only parameter that spans all tiers is the tier.global.concurrent_wait_timeout. This is the time, in seconds, that a request should wait for a dependent resource to load that is currently being loaded to fulfill a separate request.

This timeout should be adjusted to cover the expected wait time for the slowest data source in the system. If the slowest tier were the Persist Tier, a reasonable timeout may be expected to be much smaller than if the slowest tier were a Cold Storage Tier with longer transfer times.

Tier-Specific Parameters

For tier-specific parameters, the general format for defining the tier is:

1

tier.<tier_type>.<config_level>.[<config_sublevel>.]<parameter>

The tier_type is one of the five basic types of tiers:

• vram - VRAM Tier (GPU memory)
• ram - RAM Tier (Main memory)
• disk - Disk Cache Tier (Disk cache)
• persist - Persist Tier (Permanent storage)
• cold - Cold Storage Tier (Extended long-term storage)

A config_level of default applies the parameter's value to all ranks, system wide, for the respective tier. To override that default value for a specific rank, the config_level should be rank followed by the index of the rank to modify.

The only tier with a config_sublevel is the VRAM Tier, which uses all_gpus. This simply applies the default values and rank overrides to all GPU devices and acts as a placeholder for future use.

While the complete set of valid parameter names varies between tiers, most (except the Cold Storage Tier) will have the following common ones:

• limit - the maximum capacity of the tier
• high_watermark - the high watermark, as a percentage, to use for watermark-based eviction
• low_watermark - the low watermark, as a percentage, to use for watermark-based eviction

If there is no backing Cold Storage Tier, the high & low watermarks for the Persist Tier can be disabled.

For example, to set the high & low watermarks for the VRAM Tier to 90% & 50%, respectively, use:

1
2

tier.vram.default.all_gpus.high_watermark = 90
tier.vram.default.all_gpus.low_watermark = 50

To set the high & low watermarks for rank1 in the Disk Cache Tier instance disk3 to 95% & 90%, respectively, while disabling watermark-based eviction for rank2:

1
2
3
4

tier.disk3.rank1.high_watermark = 95
tier.disk3.rank1.low_watermark = 90
tier.disk3.rank2.high_watermark = 100
tier.disk3.rank2.low_watermark = 100

VRAM Tier

The default configuration assigns the VRAM Tier 95% of VRAM and high & low watermarks of 90% & 50%, respectively.

A limit of -1 reserves 95% of available video RAM for usage by the VRAM Tier. To set to a specific amount of VRAM, change this setting to be the number of bytes of VRAM to use. The high_watermark & low_watermark values are in percentages of the limit.

1
2
3

tier.vram.default.all_gpus.limit = -1
tier.vram.default.all_gpus.high_watermark = 90
tier.vram.default.all_gpus.low_watermark = 50

RAM Tier

The default configuration assigns the RAM Tier no size limit and disables watermark-based eviction.

1

tier.ram.default.limit = -1

A limit of -1 assigns no RAM size limit for usage by the RAM Tier. To set to a specific amount of RAM, change this setting to be the number of bytes of RAM to use. The high_watermark & low_watermark values are in percentages of the limit.

Disk Cache Tier

The default configuration defines no Disk Cache Tier instances.

To define a Disk Cache Tier, first ensure that a RAM Tier limit is set to a positive number that is the number of bytes of RAM to use. Without this being set, no eviction from the RAM Tier will take place, and the Disk Cache Tier will be unused.

Next, define a Disk Cache Tier instance. The instance should be identified by the keyword disk, followed by a digit, 0 - 9. The path parameter needs to be set to a directory/mount point that exists on all Kinetica hosts and has read/write access for the gpudb user. A limit of -1 assigns no disk size limit for usage by the Disk Cache Tier. To set to a specific amount of disk space to use, change this setting to be the number of bytes of disk to use. The high_watermark & low_watermark values are in percentages of the limit.

Lastly, the Disk Cache Tier can be configured to store persistent objects. If the disk backing the Disk Cache Tier is more performant than the disk backing the Persist Tier, set store_persistent_objects to true to cache persistent objects in the Disk Cache Tier. If not more performant (especially if the disks backing the Disk Cache Tier & Persist Tier are the same disk), set store_persistent_objects to false to bypass the Disk Cache Tier when writing persistent objects to disk.

For instance, to create a 500GB Disk Cache Tier instance with ID disk1, using a mount point of /opt/gpudb/diskcache, watermark-based eviction thresholds of 90% & 80%, and caching of persistent objects:

1
2
3
4
5

tier.disk1.default.path = /opt/gpudb/diskcache
tier.disk1.default.limit = 536870912000
tier.disk1.default.high_watermark = 90
tier.disk1.default.low_watermark = 80
tier.disk1.default.store_persistent_objects = true

Persist Tier

The default configuration assigns the Persist Tier no size limit and disables watermark-based eviction. It also saves database column & object data to the default Kinetica persistence directory, /opt/gpudb/persist.

1
2

tier.persist.default.path = ${gaia.persist_directory}
tier.persist.default.limit = -1

The persistence path of database objects can be modified by setting the path. A limit of -1 assigns no disk size limit for usage by the Persist Tier. To set to a specific amount of disk space, change this setting to be the number of bytes of disk to use. The high_watermark & low_watermark values are in percentages of the limit.

Cold Storage Tier

The default configuration defines no Cold Storage Tier instances.

To define a Cold Storage Tier, first ensure that the Persist Tier has a limit set to a positive number that is the number of bytes of disk space to use. Without this being set, no eviction from the Persist Tier will take place, and the Cold Storage Tier will be unused.

Next, define the Cold Storage Tier instance. The instance should be identified by the keyword cold, followed by a digit, 0 - 9. The type parameter should be set to one of the following, with a corresponding base_path:

• disk (local/network storage)
• hdfs (Hadoop File System)
• azure (Azure blob storage)
• s3 (Amazon S3)

Lastly, set the provider-specific parameters that correspond to the specified type. For local/network storage, no additional parameters are required. For remote storage, see Cold Storage Tier in the Configuration Reference for the full set of parameters.

For instance, to create a network Cold Storage Tier instance with ID cold1, using a mount point of /opt/gpudb/cold:

1
2

tier.cold1.default.type = disk
tier.cold1.default.base_path = /opt/gpudb/cold

To create an HDFS Cold Storage Tier instance with ID cold2, with a URI of hdfs://localhost:9000, an HDFS user named kinetica, a root HDFS path of /gpudb/cold, and using Kerberos with a keytab file path of /opt/gpudb/krb5.keytab:

1
2
3
4
5
6

tier.cold2.default.type = hdfs
tier.cold2.default.base_path = /gpudb/cold
tier.cold2.default.hdfs_uri = hdfs://localhost:9000
tier.cold2.default.hdfs_principal = kinetica
tier.cold2.default.hdfs_use_kerberos = true
tier.cold2.default.hdfs_kerberos_keytab = /opt/gpudb/krb5.keytab

To create a Microsoft Azure Cold Storage Tier instance with ID cold3, having a container name of kinetica, a storage account name & key of name & key, respectively, and a root path of /gpudb/cold:

1
2
3
4
5

tier.cold3.default.type = azure
tier.cold3.default.base_path = /gpudb/cold
tier.cold3.default.azure_container_name = kinetica
tier.cold3.default.azure_storage_account_name = name
tier.cold3.default.azure_storage_account_key = key

To create an Amazon S3 Cold Storage Tier instance with ID cold4, having a bucket name of kinetica, an S3 Access Key ID & Secret Access Key of key & secret, respectively, and a root S3 path of /gpudb/cold:

1
2
3
4
5

tier.cold4.default.type = s3
tier.cold4.default.base_path = /gpudb/cold
tier.cold4.default.s3_bucket_name = kinetica
tier.cold4.default.s3_aws_access_key_id = key
tier.cold4.default.s3_aws_secret_access_key = secret

Default Tier Strategy

The tier_strategy.default parameter is the default tier strategy that is applied to all tables and columns that do not have a tier strategy set explicitly during table creation or have had it subsequently altered.

A default tier strategy can include a single Disk Cache Tier instance to enable the caching to disk of all database objects, including non-persistent ones like filtered views, join views, memory-only tables, as well as those that are created temporarily to service user requests. Without a Disk Cache Tier in the default tier strategy, these other non-persistent objects will not be able to be moved out of RAM to allow for other data that may be required to support the processing of higher-priority requests.

The strategy can also include a Cold Storage Tier instance to allow database objects to be moved off of local disks, making room for other data. Without a Cold Storage Tier in the default tier strategy, each user-created persisted table will have to be designated, during creation or modification, as able to be moved off of local disks.

An example default tier strategy configuration is:

1

tier_strategy.default = VRAM 1, RAM 5, DISK1 5, PERSIST 5, COLD2

This would give all objects in the system an average default priority, while not caching any objects in VRAM and allowing transient objects to be cached on disk and persistent tables to be automatically off-loaded to cold storage.

Tier Strategy Predicate Evaluation Interval

Predicates can be used within a tier strategy to distribute & prioritize table and column data among the defined tiers. Those predicates are reevaluated at an interval to determine whether it is necessary to move data to a different tier.

The two cases where reevaluation will have an impact are:

1. The predicate contains a function that is not data-based, like NOW(), which may be used to keep current data within a given tier. For instance, to keep in RAM all records which have been modified in the last hour, the RAM Tier could have a predicate like:

   1	last_updated > NOW() - 3600

2. The predicate references a column whose values have been updated since the last evaluation interval

The parameter tier_strategy.predicate_evaluation_interval allows this interval to be specified, in minutes.