Files & Directories (KiFS)

Kinetica, via KiFS, provides support for staging files within the database for subsequent ingestion. The file structure consists of a single layer of top-level directories, with each file contained within one of those directories. File names can contain / characters to give the appearance of existing under a hierarchy of one or more sub-directories.

Directory features accessible via SQL include:

File features accessible via SQL include:

For directory permission management, see:


CREATE DIRECTORY

Creates a new KiFS directory under which files can be uploaded and used for data loading.

CREATE DIRECTORY Syntax
1
2
CREATE DIRECTORY '<directory name>'
[WITH OPTIONS (data_limit = <data limit value>)]
Parameters Description
<directory name> Name of the directory to create.
WITH OPTIONS

Optional indicator that a comma-delimited list of option/value assignments will follow. The following options are available:

Option Description
data_limit Total capacity the given directory should not exceed, in bytes; use -1 for no limit

For example, to create a directory, kdata, with no size limit:

CREATE DIRECTORY Example
1
CREATE DIRECTORY 'kdata'

To create a directory, fs_limited, with a 1,000,000 byte maximum capacity:

CREATE DIRECTORY with Limit Example
1
2
CREATE DIRECTORY 'fs_limited'
WITH OPTIONS (data_limit = 1000000)

ALTER DIRECTORY

Alters the configuration of an existing KiFS directory.

The following facet of a directory can be altered:

Set Data Limit

A directory can have its data limit modified.

Set Directory Data Limit Syntax
1
2
ALTER DIRECTORY '<directory name>'
SET data_limit = <data limit value>
Parameters Description
<directory name> Name of the directory to alter
<data limit value> Total capacity the given directory should not exceed, in bytes; use -1 for no limit

For example, to set the maximum capacity of a directory, fs_limited, to 10,000,000 bytes:

ALTER DIRECTORY SET Data Limit Example
1
2
ALTER DIRECTORY 'fs_limited'
SET data_limit = 10000000

LIST DIRECTORY

Outputs detail about one or all KiFS directories.

LIST DIRECTORY Syntax
1
<LIST|SHOW|DESC[RIBE]> DIRECTOR[Y|IES] < '<directory name>' | * >
Parameters Description
<directory name> Name of the existing directory for which detail will be output. Use * instead to output detail of all directories.

Note

The response to LIST DIRECTORY is a six-column result set:

  • DIRECTORY_NAME - name of the directory
  • CREATED_BY - user ID of the user who created the directory
  • CREATION_TIME - date/time at which the directory was created
  • DATA_USAGE - amount of used file storage within the directory, in bytes
  • DATA_LIMIT - total amount of file storage within the directory, in bytes
  • PERMISSION - highest level of access to the directory the command issuer has

For example, to output the detail about a directory, kdata:

LIST DIRECTORY Example
1
LIST DIRECTORY 'kdata'

To output the detail for all directories:

LIST DIRECTORIES (All Directories) Example
1
LIST DIRECTORIES *

DOWNLOAD DIRECTORY

Downloads all files under a directory in KiFS into a local directory.

Note

This command is only available through KiSQL or database clients configured with the Kinetica JDBC driver.

DOWNLOAD DIRECTORY Syntax
1
2
DOWNLOAD [DIRECTORY] <kifs directory>
INTO '<local directory>'
Parameters Description
DIRECTORY Optional keyword for clarity.
<kifs directory> KiFS directory whose contained files will be downloaded to the local file system.
<local directory> Local directory path into which files will be downloaded.

For example, to download all files in a KiFS directory to a local directory, /tmp:

DOWNLOAD DIRECTORY Example
1
2
DOWNLOAD DIRECTORY 'kdata'
INTO '/tmp'

DROP DIRECTORY

Removes an existing KiFS directory.

DROP DIRECTORY Syntax
1
2
DROP DIRECTORY [IF EXISTS] '<directory name>'
[WITH OPTIONS ('<option name>' = '<option value>'[,...])]
Parameters Description
<directory name> Name of the directory to remove.
IF EXISTS Optional error-suppression clause; if specified, no error will be returned if the given directory does not exist.
WITH OPTIONS

Optional indicator that a comma-delimited list of option/value assignments will follow.

Option Description
recursive

Whether to delete all files contained within the directory:

  • false (default) - only drop the directory if empty; return an error if the directory contains files
  • true - drop the directory and all files in it

For example, to delete a directory, kbackup:

DROP DIRECTORY Example
1
DROP DIRECTORY 'kbackup'

To delete a directory, kdata, and all files in it, suppressing the error if it doesn't exist:

DROP DIRECTORY with Files Example
1
2
DROP DIRECTORY IF EXISTS 'kdata'
WITH OPTIONS ('recursive' = 'true')

UPLOAD FILE

Uploads file(s) local to the database client into KiFS, within the given directory.

Note

This command is only available through KiSQL or database clients configured with the Kinetica JDBC driver.

UPLOAD FILE Syntax
1
2
UPLOAD FILE[S] <file path(s)>
INTO '<kifs path>'
Parameters Description
<file path(s)>

Local path(s) of the file(s) to upload to KiFS, as a comma-separated list of single-quoted file paths; wildcards (*) can be used to specify a group of files.

File Path Outcome
'archive/*.csv' Upload all files under the archive directory with a csv extension
'data/q1.csv','data/q2.csv' Upload q1.csv & q2.csv that reside under the data directory
<kifs path>

Path in KiFS; this can be either the name of a directory or a directory followed by virtual subdirectories separated by / characters.

For example, uploading a file named 2021.q1.csv to:

  • 'data' - will upload it to the data directory and make it available as data/2021.q1.csv
  • 'data/sales/Q1' - will upload it to the data directory and make it available as data/sales/Q1/2021.q1.csv

Note

Only the directory specified within this path must exist for the upload to succeed. Any virtual subdirectories added to this path do not need to exist before uploading.

For example, to upload a file to a directory, kdata:

UPLOAD FILE Example
1
2
UPLOAD FILE '../data/products.csv'
INTO 'kdata'

To upload several files to a directory, kdata:

UPLOAD FILES Example
1
2
UPLOAD FILES '../data/p*.ssv', '../data/employee.*'
INTO 'kdata'

UPLOAD URL

Uploads file(s) from one or more URLs into KiFS, in one or more directories. There should be a corresponding KiFS full file path specified for each URL specified--the KiFS file name will not be derived from any part of the given URL.

UPLOAD URL Syntax
1
2
UPLOAD URL <url(s)>
INTO <kifs path(s)>
Parameters Description
<url(s)> URL(s) of the file(s) to upload to KiFS, as a comma-separated list of single-quoted URL(s).
<kifs path(s)>

File path(s) in KiFS to upload the file(s) at the URL(s) to, as a comma-separated list of single-quoted KiFS full file paths.

Note

Only the directory specified within this path must exist for the upload to succeed. Any virtual subdirectories within this path do not need to exist before uploading.

For example, to upload the Kinetica JDBC Driver in GitHub to file kinetica-jdbc-fullshaded.jar under directory drivers:

UPLOAD URL Example
1
2
UPLOAD URL 'https://github.com/kineticadb/kinetica-client-jdbc/raw/master/kinetica-jdbc-7.1.9.9-fullshaded.jar'
INTO 'drivers/kinetica-jdbc-fullshaded.jar';

To upload the Kinetica JDBC Driver and the corresponding change log in GitHub to files kjdbc-7.1.jar & kjdbc-CHANGELOG.md, respectively, under directory drivers:

UPLOAD URLs Example
1
2
3
4
5
6
UPLOAD URL
      'https://github.com/kineticadb/kinetica-client-jdbc/raw/master/kinetica-jdbc-7.1.9.9-fullshaded.jar',
      'https://github.com/kineticadb/kinetica-client-jdbc/raw/master/CHANGELOG.md'
INTO
      'drivers/kjdbc-7.1.jar',
      'drivers/kjdbc-CHANGELOG.md';

LIST FILE

Outputs detail about one or all KiFS files.

SHOW FILE Syntax
1
<LIST|SHOW|DESC[RIBE]> FILE[S] <kifs path(s)>
Parameters Description
<kifs path(s)>

Name(s) of existing directories and/or files for which detail will be output, as a comma-separated list of single-quoted paths.

Path Outcome
'archive' Output detail of all files under the KiFS directory archive.
'data/q1.csv','data/q2.csv' Output detail of q1.csv & q2.csv that reside under the data directory.

Note

The response to LIST FILE is a four-column result set:

  • FILE_NAME - name of the directory
  • SIZE - size of the file in bytes
  • CREATED_BY - user ID of the user who created the file
  • CREATION_TIME - date/time at which the file was created

For example, to output a listing of all files under the KiFS directory, kdata:

LIST FILES (Directory) Example
1
LIST FILES 'kdata'

To output a listing of specific files under the KiFS directory, kdata:

LIST FILES (Specific Files) Example
1
LIST FILES 'kdata/products.csv', 'kdata/employee.parquet'

DOWNLOAD FILE

Downloads file(s) from KiFS into a local directory.

Note

This command is only available through KiSQL or database clients configured with the Kinetica JDBC driver.

DOWNLOAD FILE Syntax
1
2
DOWNLOAD [FILE[S]] <kifs file path(s)>
INTO '<local directory>'
Parameters Description
FILE[S] Optional keyword for clarity.
<kifs file path(s)>

KiFS file(s) to download to the local file system, as a comma-separated list of single-quoted file paths.

File Path Outcome
'archive/products.csv' Download archive/products.csv
'data/q1.csv','data/q2.csv' Download q1.csv & q2.csv that reside under the data directory
<local directory> Local directory path into which files will be downloaded.

For example, to download a KiFS file to a local directory, /tmp:

DOWNLOAD FILE Example
1
2
DOWNLOAD FILE 'kdata/products.csv'
INTO '/tmp'

To download several KiFS files to a local directory, /tmp:

DOWNLOAD FILES Example
1
2
DOWNLOAD FILES 'kdata/products.ssv', 'kdata/employee.parquet'
INTO '/tmp'

DROP FILE

Deletes an existing KiFS file.

DROP FILE Syntax
1
DROP FILE[S] [IF EXIST[S]] <kifs file path(s)>
Parameters Description
<kifs file path(s)> KiFS file(s) to delete, as a comma-separated list of single-quoted file paths.
IF EXIST[S] Optional error-suppression clause; if specified, no error will be returned if the given file(s) do not exist.

For example, to delete a KiFS file:

DROP FILE Example
1
DROP FILE 'kdata/products.csv'

To delete several KiFS files, suppressing the error if any don't exist:

DROP FILES Example
1
DROP FILES IF EXIST 'kdata/products.csv', 'kdata/employee.parquet'