Data Sources

A data source is reference object for a data set that is external to the database. It consists of the location & connection information to that external source, but doesn't hold the names of any specific data sets/files within that source. A data source can make use of a credential object for storing remote authentication information.

A data source name must adhere to the standard naming criteria. Each data source exists within a schema and follows the standard name resolution rules for tables.

The following data source providers are supported:

  • Azure (Microsoft blob storage)
  • GCS (Google Cloud Storage)
  • HDFS (Apache Hadoop Distributed File System)
  • JDBC (Java Database Connectivity, using a user-supplied driver or one of the drivers on the supported list)
  • Kafka (streaming feed)
    • Apache
    • Confluent
  • S3 (Amazon S3 Bucket)

Note

The following default hosts are used for Azure, GCS, & S3, but can be overridden in the location parameter:

  • Azure: <service_account_name>.blob.core.windows.net
  • GCS: storage.googleapis.com
  • S3: <region>.amazonaws.com

Data sources perform no function by themselves, but act as proxies for accessing external data when referenced in certain database operations. The following can make use of data sources:

Individual files within a data source need to be identified when the data source is referenced within these calls.

Note

The data source will be validated upon creation, by default, and will fail to be created if an authorized connection cannot be established.


Managing Data Sources

A data source can be managed using the following API endpoint calls. For managing data sources in SQL, see CREATE DATA SOURCE.

API CallDescription
/create/datasourceCreates a data source, given a location and connection information
/alter/datasourceModifies the properties of a data source, validating the new connection
/drop/datasourceRemoves the data source reference from the database; will not modify the external source data
/show/datasourceOutputs the data source properties; passwords are redacted
/grant/permission/datasourceGrants the permission for a user to connect to a data source
/revoke/permission/datasourceRevokes the permission for a user to connect to a data source

Creating a Data Source

To create a data source, kin_ds, that connects to an Amazon S3 bucket, kinetica_ds, in the US East (N. Virginia) region, in Python:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = 'kin_ds',
    location = 's3',
    user_name = aws_id,
    password = aws_key,
    options = {
        's3_bucket_name': 'kinetica-ds',
        's3_region': 'us-east-1'
    }
)

Important

For Amazon S3 connections, the user_name & password parameters refer to the AWS Access ID & Key, respectively.


Provider-Specific Syntax

Several authentication schemes across multiple providers are supported.


Azure

Credential 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'azure[://<host>]',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        'azure_container_name': '<azure container name>'
    }
)
Managed Credentials 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'azure[://<host>]',
    options = {
        'use_managed_credentials': 'true',
        'azure_storage_account_name': '<azure storage account name>',
        'azure_container_name': '<azure container name>',
        'azure_tenant_id': '<azure tenant id>'
    }
)
Public (No Auth) 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'azure[://<host>]',
    user_name = '<azure storage account name>',
    password = '',
    options = {
        'azure_container_name': '<azure container name>'
    }
)
Password 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'azure[://<host>]',
    user_name = '<azure storage account name>',
    password = '<azure storage account key>',
    options = {
        'azure_container_name': '<azure container name>'
    }
)
SAS Token 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'azure[://<host>]',
    user_name = '<azure storage account name>',
    password = '',
    options = {
        'azure_sas_token': '<azure sas token>',
        'azure_container_name': '<azure container name>'
    }
)
Active Directory 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'azure[://<host>]',
    user_name = '<ad client id>',
    password = '<ad client secret key>',
    options = {
        'azure_storage_account_name': '<azure storage account name>',
        'azure_container_name': '<azure container name>',
        'azure_tenant_id': '<azure tenant id>'
    }
)

GCS

Credential 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'gcs[://<host>]',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        ['gcs_project_id': '<gcs project id>',]
        'gcs_bucket_name': '<gcs bucket name>'
    }
)
Managed Credentials 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'gcs[://<host>]',
    user_name = '',
    password = '',
    options = {
        'use_managed_credentials': 'true',
        ['gcs_project_id': '<gcs project id>',]
        'gcs_bucket_name': '<gcs bucket name>'
    }
)
Public (No Auth) 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'gcs[://<host>]',
    user_name = '',
    password = '',
    options = {
        ['gcs_project_id': '<gcs project id>',]
        'gcs_bucket_name': '<gcs bucket name>'
    }
)
User ID & Key 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'gcs[://<host>]',
    user_name = '<gcs account id>',
    password = '<gcs account private key>',
    options = {
        ['gcs_project_id': '<gcs project id>',]
        'gcs_bucket_name': '<gcs bucket name>'
    }
)
JSON Key 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'gcs[://<host>]',
    user_name = '',
    password = '',
    options = {
        'gcs_service_account_keys': '<gcs account json key text>',
        ['gcs_project_id': '<gcs project id>',]
        'gcs_bucket_name': '<gcs bucket name>'
    }
)

HDFS

Credential 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'hdfs://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>'
    }
)
Password 
1
2
3
4
5
6
7

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'hdfs://<host>:<port>',
    user_name = '<hdfs username>',
    password = '<hdfs password>',
    options = {}
)
Kerberos Token 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'hdfs://<host>:<port>',
    user_name = '<hdfs username>',
    password = '',
    options = {
        'hdfs_use_kerberos': 'true'
    }
)
Kerberos Keytab 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'hdfs://<host>:<port>',
    user_name = '<hdfs username>',
    password = '',
    options = {
        'hdfs_kerberos_keytab': 'kifs://<keytab file/path>'
    }
)

JDBC

Credential 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = '<jdbc url>',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        'jdbc_driver_class_name': '<jdbc driver class full path>',
        'jdbc_driver_jar_path': 'kifs://<jdbc driver jar path>'
    }
)
Password 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = '<jdbc url>',
    user_name = '<jdbc username>',
    password = '<jdbc password>',
    options = {
        'jdbc_driver_class_name': '<jdbc driver class full path>',
        'jdbc_driver_jar_path': 'kifs://<jdbc driver jar path>'
    }
)

Kafka (Apache)

Note

The location can be a comma-delimited list of Kafka URLs to be used for high-availability; only one of which will be streamed from at any given time.

Credential 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'kafka://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        'kafka_topic_name': '<kafka topic name>'
    }
)
Credential w/ Schema Registry 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'kafka://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        'kafka_topic_name': '<kafka topic name>',
        'schema_registry_credential': '[<sr credential schema name>.]<sr credential name>',
        'schema_registry_location': '<schema registry url>'
    }
)
Public (No Auth) 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'kafka://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'kafka_topic_name': '<kafka topic name>'
    }
)

Kafka (Confluent)

Note

The location can be a comma-delimited list of Kafka URLs to be used for high-availability; only one of which will be streamed from at any given time.

Credential 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'confluent://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        'kafka_topic_name': '<kafka topic name>'
    }
)
Credential w/ Schema Registry 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'confluent://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        'kafka_topic_name': '<kafka topic name>',
        'schema_registry_credential': '[<sr credential schema name>.]<sr credential name>',
        'schema_registry_location': '<schema registry url>'
    }
)
Public (No Auth) 
1
2
3
4
5
6
7
8
9

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 'confluent://<host>:<port>',
    user_name = '',
    password = '',
    options = {
        'kafka_topic_name': '<kafka topic name>'
    }
)

S3

Credential 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 's3[://<host>]',
    user_name = '',
    password = '',
    options = {
        'credential': '[<credential schema name>.]<credential name>',
        's3_bucket_name': '<aws s3 bucket name>',
        's3_region': '<aws s3 region>'
    }
)
Managed Credentials 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 's3[://<host>]',
    user_name = '',
    password = '',
    options = {
        'use_managed_credentials': 'true',
        's3_bucket_name': '<aws s3 bucket name>',
        's3_region': '<aws s3 region>'
    }
)
Public (No Auth) 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 's3[://<host>]',
    user_name = '',
    password = '',
    options = {
        's3_bucket_name': '<aws s3 bucket name>',
        's3_region': '<aws s3 region>'
    }
)
Access Key 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 's3[://<host>]',
    user_name = '<aws access key id>',
    password = '<aws secret access key>',
    options = {
        's3_bucket_name': '<aws s3 bucket name>',
        's3_region': '<aws s3 region>'
    }
)
IAM Role 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

kinetica.create_datasource(
    name = '[<data source schema name>.]<data source name>',
    location = 's3[://<host>]',
    user_name = '<aws access key id>',
    password = '<aws secret access key>',
    options = {
        's3_bucket_name': '<aws s3 bucket name>',
        's3_region': '<aws s3 region>',
        's3_aws_role_arn': '<amazon resource name>'
    }
)

Limitations

  • Azure anonymous data sources are only supported when both the container and the contained objects allow anonymous access.
  • HDFS systems with wire encryption are not supported.
  • Kafka data sources require an associated credential object for authentication.