Configuration Reference¶

Kinetica configuration consists of several user-modifiable system parameters present in /opt/gpudb/core/etc/gpudb.conf that are used to set up and tune the system.

Network setup¶

Configuration parameter	default value	description
head_ip_address	127.0.0.1	Head HTTP server IP address. Set to the publicly accessible IP address of the first process, rank 0.
head_port	9191	Head HTTP server port to use for 'head_ip_address'.
use_https	false	Set to true to use HTTPS; if true then https_key_file and https_cert_file must be provided
https_key_file https_cert_file		Files containing the SSL private Key and the SSL certificate for. If required, a self signed certificate (expires after 10 years) can be generated via the command openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
http_allow_origin		Value to return via Access-Control-Allow-Origin HTTP header (for Cross-Origin Resource Sharing). Set to empty to not return the header and disallow CORS.
enable_httpd_proxy	false	Start a httpd server as a proxy to handle LDAP and/or Kerberos authentication. Each host will run a httpd server and access to each rank is available through http://host:8082/gpudb-1, where port 8082 is defined by 'httpd_proxy_port'. NOTE: HTTPd external endpoints are not affected by the use_https parameter above. If you wish to enable https on HTTPd, you must edit the /opt/gpudb/httpd/conf/httpd.conf and setup https as per the Apache HTTPd documentation at https://httpd.apache.org/docs/2.2/
httpd_proxy_port	8082	TCP port that the httpd auth proxy server will listen on if 'enable_httpd_proxy' is true.
rank0_ip_address	${gaia.rank0.host}	Internal use IP address of the head HTTP server, rank 0. Set to either a second internal network accessible by all ranks or to ${gaia.head_ip_address}.
trigger_port	9001	Trigger ZMQ publisher server port (-1 to disable), uses the 'head_ip_address' interface.
set_monitor_port	9002	Set monitor ZMQ publisher server port (-1 to disable), uses the 'head_ip_address' interface.
enable_odbc_connector	true	Enable ODBC connector
enable_reveal	true	Enable Reveal runtime
global_manager_port_one	5552	Internal communication ports
global_manager_pub_port	5553	Host manager synchronization port
host_manager_http_port	9300	Http port for web portal of the host manager
enable_worker_http_servers	false	Enable worker HTTP servers, each process runs its own server for direct ingest.
rank1.worker_http_server_port rank2.worker_http_server_port	9192 9193	Optionally, specify the worker HTTP server ports. The default is to use (head_port + rank#) for each worker process where rank number is from 1 to number of ranks in 'rankN.host' below.
rank0.host rank1.host rank2.host	127.0.0.1 127.0.0.1 127.0.0.1	Specify the hosts to run each rank worker process in the cluster on. For a single machine system you may use '127.0.0.1', but if using two or more machines you must specify a hostname or IP address for each rank that is accessible to the other ranks. See also 'head_ip_address' and 'rank0_ip_address'.
rank0.communicator_port	6555	Specify the TCP ports each rank will use to communicate with each other. If the port for any rankN is not specified the port will be assigned to rank0.communicator_port + N.
compress_network_data	false	rank1.communicator_port = 6556 rank2.communicator_port = 6557 Enables compression of inter-node network data transfers.
communicator_type	ZMQ	Communicator type (either MPI or ZMQ)

Security¶

Configuration parameter	default value	description
require_authentication	false	Require authentication.
enable_authorization	false	Enable authorization checks.
min_password_length	0	Minimum password length.
enable_external_authentication	false	Enable external (LDAP, Kerberos, etc.) authentication. User IDs of externally-authenticated users must be passed in via the REMOTE_USER HTTP header from the authentication proxy. May be used in conjuntion with the enable_httpd_proxy setting above for an integrated external authentication solution. IMPORTANT: DO NOT ENABLE unless external access to GPUdb ports has been blocked via firewall AND the authentication proxy is configured to block REMOTE_USER HTTP headers passed in from clients.
auto_create_external_users	false	Automatically create accounts for externally-authenticated users. If enable_external_authentication is false, this setting has no effect. Note that accounts are not automatically deleted if users are removed from the external authentication provider and will be orphaned.
auto_grant_external_roles	false	Automatically add roles passed in via the KINETICA_ROLES HTTP header to externally-authenticated users. Specified roles that do not exist are ignored. If enable_external_authentication is false, this setting has no effect. IMPORTANT: DO NOT ENABLE unless the authentication proxy is configured to block KINETICA_ROLES HTTP headers passed in from clients.
auto_revoke_external_roles		Comma-separated list of roles to revoke from externally-authenticated users prior to granting roles passed in via the KINETICA_ROLES HTTP header, or * to revoke all roles. Preceding a role name with an ! overrides the revocation (e.g. *,!foo revokes all roles except foo). Leave blank to disable. If either enable_external_authentication or auto_grant_external_roles is false, this setting has no effect.

Auditing¶

Configuration parameter	default value	description
enable_audit	false	This section controls the request auditor, which will audit all requests received by the server in full or in part based on the settings below. The output location of the audited requests is controlled via settings in the Auditing section of gpudb_logger.conf. enable_audit: Controls whether request auditing is enabled. If set to true, the following information is audited for every request: Job ID, URI, User, and Client Address. The settings below control whether additional information about each request is also audited. If set to false, all auditing is disabled.
audit_headers	false	audit_headers: Controls whether HTTP headers are audited for each request. If enable_audit is false this setting has no effect.
audit_body	false	audit_body: Controls whether the body of each request is audited (in JSON format). If enable_audit is false this setting has no effect. Note: For requests that insert data records, this setting does not control the auditing of the records being inserted, only the rest of the request body; see audit_data below to control this.
audit_data	false	audit_data: Controls whether records being inserted are audited (in JSON format) for requests that insert data records. If either enable_audit or audit_body is false, this setting has no effect. Note: Enabling this setting during bulk ingestion of data will rapidly produce very large audit logs and may cause disk space exhaustion; use with caution.
lock_audit	false	lock_audit: Controls whether the above audit settings can be altered at runtime via the /alter/system/properties endpoint. In a secure environment where auditing is required at all times, this should be set to true to lock the settings to what is set in this file.

Licensing¶

Configuration parameter	default value	description
license_key		The license key to authorize running.

Process and thread configuration¶

Configuration parameter	default value	description
min_http_threads	2	Set min number of web server threads to spawn. (default: 2)
max_http_threads	64	Set max number of web server threads to spawn. (default: 64)
sm_omp_threads	2	Set the number of parallel jobs to create for multi-child set calculations. Use -1 to use the max number of threads (not recommended).
kernel_omp_threads	4	Set the number of parallel calculation threads to use for data processing. Use -1 to use the max number of threads (not recommended).
max_tbb_threads_per_rank	-1	Set the maximum number of threads (both workers and masters) to be passed to TBB on initialization. Generally speaking, max_tbb_threads_per_rank - 1 TBB workers will be created.
toms_per_rank	4	Set the number of TOMs per rank, the number of data container shards per rank.
tps_per_tom	4	Set the number of TaskProcessors per TOM, CPU data processors.
tcs_per_tom	4	Set the number of TaskCalculators per TOM, GPU data processors.
persistence_replication_factor	1	Set the number of Backup TOMs, per TOM.

Hardware configuration¶

Configuration parameter	default value	description
rank0.gpu	0	Specify the GPU to use for all calculations on the HTTP server node, rank 0. Note that the rank0 GPU may be shared with another rank.
rank1.taskcalc_gpu rank2.taskcalc_gpu		Set GPU devices for each worker rank's TaskCalculators, see 'tcs_per_tom'. If no gpus are specified, each rank's TaskCalculators will share the same GPU and each rank will round-robin the available gpus on the system. Ideally, each rank should use a single specified GPU to allow data caching. Add rankN.taskcalc_gpu as needed, where N ranges from 1 to the highest index in 'rankN.host' For the example below the successively created taskcalcs use the 0,1,2 GPU devices respectively rank1.taskcalc_gpu = 0 1 2
rank0.numa_node		Set the head HTTP rank0 numa node(s). If left empty there will be no thread affinity or preferred memory node. The node list may be either a single node number or a range, E.G. 1-5,7,10. If there will be many simultaneous users, specify as many nodes as possible that won't overlap the rank1+ worker numa nodes that the GPUs are on. If there will be few simultaneous users and WMS speed is important, choose the numa node the 'rank0.gpu' is on.
rank1.base_numa_node rank2.base_numa_node		Set each worker rank's preferred base numa node for CPU affinity and memory allocation. The 'rankN.base_numa_node' is the node or nodes that non-data intensive threads will run in. These nodes do not have to be the same numa nodes that the GPU specified by the corresponding 'rankN.taskcalc_gpu' is on for best performance, though they should be relatively near to their 'rankN.data_numa_node'. There will be no CPU thread affinity or preferred node for memory allocation if not specified or left empty. The node list may be a single node number or a range, E.G. 1-5,7,10.
rank1.data_numa_node rank2.data_numa_node		Set each worker rank's preferred data numa node for CPU affinity and memory allocation. The 'rankN.data_numa_node' is the node or nodes that data intensive threads will run in and should be set to the same numa node that the GPU specified by the corresponding 'rankN.taskcalc_gpu' is on for best performance. If the 'rankN.taskcalc_gpu' is specified the 'rankN.data_numa_node' will be automatically set to the node the GPU is attached to, otherwise there will be no CPU thread affinity or preferred node for memory allocation if not specified or left empty. The node list may be a single node number or a range, E.G. 1-5,7,10.

General configuration¶

Configuration parameter	default value	description
enable_job_manager	true	Enable JobManager to enforce processing of requests in the order received
protected_sets	MASTER,_MASTER,_DATASOURCE	Tables with these names will not be deleted (comma separated).
default_ttl	20	Time-to-live in minutes of non-protected tables before they are deleted.
memory_ttl	9999999	Time in minutes for an unused data tables to stay in memory.
disable_clear_all	false	Disallow the 'clear/table' command to clear all tables.
point_render_threshold	100000	Threshold number of points (per-TOM) at which point rendering switches to fast mode.
symbology_render_threshold	10000	Threshold for the number of points (per-TOM) after which symbology rendering falls back to regular rendering
max_heatmap_size	3072	Maximum size (in pixels) heatmap that can be generated. This reserves MHSMHS8 bytes of GPU memory at rank0
symbol_resolution	100	The image width/height (in pixels) of svg symbols cached in the OpenGL symbol cache.
symbol_texture_size	4000	The width/height (in pixels) of an OpenGL texture which caches symbol images for OpenGL rendering.
enable_opengl_renderer	true	Enable (if true) HW-accelerated OpenGL renderer instead of Cairo renderer (software renderer).
enable_pinned_memory	false	Enable (if true) or disable pinned memory; disabling pinned memory uses malloc and will result in slower transfer times to the GPU, but allows more system memory to be used.
pinned_memory_pool_size	250000000	Size in bytes of the pinned memory pool per-rank process to speed copying data to the GPU. Set to 0 to disable and 'enable_pinned_memory' must be 'false' if enabled.
memory_allocation_limit_mb	0	Maximum host memory allocation limit per rank, 0 means no limit. Specify the default maximum allowed memory allocation size for each rank and optionally override the limit for individual ranks that are running on machines with either more or less memory than the others. This is hard limit and requests requiring large allocations will be refused. Allow for at least 1GB of extra memory per rank and account for the number of other GPUdb ranks and processes running on each machine. Example, each rank will allocate ~128GB of memory, but rank three and four will allocate up to ~256GB. memory_allocation_limit_mb = 128000 rank3.memory_allocation_limit_mb = 256000 rank4.memory_allocation_limit_mb = 256000
concurrent_kernel_execution	false	Enable (if true) multiple kernels to run concurrently on the same GPU
max_concurrent_kernels	4	Maximum number of kernels that can be running at the same time on a given GPU. Set to 0 for no limit. Only takes effect if concurrent_kernel_execution = true
enable_gpu_allocator	true	Enable (if true) the GPU memory pre-allocator
max_gpu_memory	-1	Maximum amount of GPU memory to allocate in bytes, set to -1 to allocate the maximum. Only works if enable_gpu_allocator is true.
fast_polling	false	Enable (if true) fast polling on the worker ranks (at the costof 100% cpu usage)
max_get_records_size	20000	Maximum number of records that data retrieval requests such as /get/records and /aggregate/groupby will return per request.
request_timeout	20	Timeout (in minutes) for filter-type requests
max_query_temp_memory	64000000000	Maximum amount of temporary memory (in bytes) a query can allocate (per-tom); if more is needed the query will fail
on_startup_script		Set an optional executable command that will be run once when Kinetica is ready for client requests. This can be used to perform any initialization logic that needs to be run before clients connect. It will be run as the gpudb user, so you must ensure that any required permissions are set on the file to allow it to be executed. If the command cannot be executed or returns a non zero error code, then Kinetica will be stopped. Output from the startup script will be logged to /opt/gpudb/core/logs/gpudb-on-start.log (and its dated relatives). The gpudb_env.sh script is run directly before the command, so the path will be set to include the supplied python runtime. Example: on_startup_script = /home/gpudb/on-start.sh param1 param2 ...
timeout_startup_subsystem	60	Timeout (in seconds) to wait for each database subsystem to startup. Subsystems include the ODBC, stats and httpd servers, as well as external text-search ranks.
timeout_shutdown_subsystem	20	Timeout (in seconds) to wait for each database subsystem to exit gracefully before it is force killed.
timeout_shutdown_rank	300	Timeout (in seconds) to wait for a rank to exit gracefully before it is force killed. Machines with slow disk drives may require longer times and data may be lost if drive is not responsive.

Tomcat Configuration¶

Configuration parameter	default value	description
enable_tomcat	true

Text search configuration¶

Configuration parameter	default value	description
enable_text_search	true	Enable text search.
use_external_text_server	false	Use an external text server instead of an internal one. Be sure to update the rankN.text_index_address and rankN.text_search_address params below.
text_indices_per_tom	2	Number of text indices to start for each rank
text_searcher_refresh_interval	20	Searcher refresh intervals - specifies the maximum delay (in seconds) between writing to the text search index and being able to search for the value just written. A value of 0 insures that writes to the index are immediately available to be searched. A more nominal value of 100 should improve ingest speed at the cost of some delay in being able to text search newly added values.
rank1.text_index_address rank2.text_index_address	ipc:///tmp/gpudb-text-index-1 ipc:///tmp/gpudb-text-index-2	External text server addresses to use if 'use_external_text_server = true'. Specify one for each worker rank N, where N ranges from 1 to highest index in 'rankN.host'. Add the appropriate number of rankN.text_index_address for each worker rank as needed. The addresses can be a fully qualified TCP address:port for remote servers or an IPC address for local text index servers. If no addresses are specified the text index servers will use IPC and be started on the machine where the rank is running as shown in the IPC example below. You should either specify all addresses or none to get the defaults. Example for remote or TCP servers: rank1.text_index_address = tcp://127.0.0.1:4000 rank2.text_index_address = tcp://127.0.0.1:4001 ... up to rank[N].text_index_address = ... Example for local IPC servers: rank1.text_index_address = ipc:///tmp/gpudb-text-index-1 rank2.text_index_address = ipc:///tmp/gpudb-text-index-2 ... up to rank[N].text_index_address = ... Where '/tmp/gpudb-text-index-1' is the name of the socket file to create.

Data persistence storage configuration and directories¶

Configuration parameter	default value	description
persist_directory	/opt/gpudb/persist	Specify a base directory to store persistence data files.
data_directory	${gaia.persist_directory}	Base directory to store data vectors.
object_directory	${gaia.persist_directory}	Base directory to store added objects.
sms_directory	${gaia.persist_directory}	Base directory to store hashed strings.
text_index_directory	${gaia.persist_directory}	Base directory to store the text search index.
temp_directory	/tmp	Directory for GPUdb to use to store temporary files. Must be a fully qualified path, have at least 100Mb of free space, and execute permission.
persist_encryption_pass_command		Path to script that echos the password for mounting encrypted persistence dirs. Note that all directories must be encrypted with the same password. Leave blank if the persistence directories are not encrypted or if the directories will be mounted before GPUdb starts.
persist_sync	false	Synchronous persistence file writing instead of asynchronous writing.
persist_sync_time	5	Force syncing the persistence files every N minutes if out of sync. Note that files are always opportunistically saved, this simply enforces a maximum time a file can be out of date. Set to a very high number to disable.
synchronous_update	true	If true then updates are applied to IndexDB synchronously (slower) If false then updates are asynchronous (faster but less safe)
load_vectors_on_start	always	When starting, 'always' load all the data into memory before accepting requests, load the necessary data to start, but load the remainder lazily, or only load the data 'on_demand' as requests use it. load_vectors_on_start = 'always' / 'lazy' / 'on_demand'
indexdb_toc_size	1000000	Table of contents size for IndexedDb object file store
indexdb_max_open_files	128	Maximum number of open files for IndexedDb object file store
sms_max_open_files	128	Maximum number of open files (per-TOM) for the SMS (string) store
chunk_size	8000000	Chunk size (0 disables chunking)
max_join_query_dimensions	2	Max join dimensions is the maximum number of tables in a joined table that can be accessed by a query and are not equated by a foreign-key to primary-key equality predicate.
execution_mode	device	Determines whether to execute kernels on host (cpu) or device (gpu). Possible values are: 'default' : engine decides 'host' : execute only host 'device' : execute only device <integer> : execute on the host if chunked column has equal or fewer rows, otherwise execute on device.

Stats logging configuration¶

Configuration parameter

default value

description

enable_stats_server

true

Run a stats server to collect information about GPUdb and machines it is running on.

stats_server_ip_address

stats_server_port

${gaia.rank0_ip_address}

2003

Stats server ip address (run on head node) default port is 2003

stats_server_namespace

gpudb

Status server namepace - should be a machine identifier

Procs¶

Configuration parameter	default value	description
enable_procs	false	Enable procs
proc_directory	${gaia.temp_directory}	Directory where proc files are stored at runtime. Must be a fully qualified path with execute permission. If not specified, temp_directory will be used.
proc_data_directory	${gaia.temp_directory}	Directory where data transferred to and from procs is written. Must be a fully qualified path with sufficient free space for required volume of data. If not specified, temp_directory will be used.

KiFS¶

Configuration parameter	default value	description
enable_kifs	false	Enable access to the KiFS file system from within procs. This will create a 'filesystem' collection and an internal 'kifs' user in Kinetica and mount the file system at the mount point specified below.
kifs_mount_point	/opt/gpudb/kifs	Parent directory of the mount point for the KiFS file system. Must be a fully qualified path. The actual mount point will be a subdirectory 'mount' below this directory.

HA¶

Configuration parameter

default value

description

enable_ha

false

At the moment, this section is simply to provide information about if and where HA is running, so that the gpudb startup script can properly create proxy-pass entries for httpd when both HA and httpd are enabled. Enable HA

ha_proc_ip_address

ha_proc_port