Python UDF API

Dependencies

To begin writing Python UDFs, access to the Kinetica Python UDF API is required.

The API can be downloaded from the Python UDF API repo on GitHub. After downloading, see the README.md in the UDF API directory created for further setup instructions.

Instructions for installing & configuring the API can be found under UDF API Installation below.

The Python UDF API consists of one file, kinetica_proc.py. This needs to be included in the UDF source code and added to the PYTHONPATH. There are no external dependencies beyond the Python standard library.

To take advantage of GPU processing within a UDF, the CUDA Toolkit must be downloaded & installed from the NVIDIA Developer Zone.

Also, any Python packages the UDF may need to use should be installed on the development platform and be made available within the Kinetica cluster's UDF environment using function environments.

1
2
3
4

CREATE FUNCTION ENVIRONMENT udfe_st;

ALTER FUNCTION ENVIRONMENT udfe_st
INSTALL PACKAGE 'urllib3==1.26.18 sentence-transformers'

1
2
3
4
5
6
7
8
9

PROC_ENV_NAME = 'udfe_st'

kinetica.create_environment(environment_name = PROC_ENV_NAME)

kinetica.alter_environment(
    environment_name = PROC_ENV_NAME, 
    action = "install_package", 
    value = "urllib3==1.26.18 sentence-transformers"
)

1
2
3
4
5
6

CREATE FUNCTION udf_st
MODE = 'distributed'
RUN_COMMAND = 'python'
RUN_COMMAND_ARGS = 'udf/udf_st.py'
FILE PATHS 'kifs://udf/udf_st.py'
WITH OPTIONS (SET_ENVIRONMENT = 'udfe_st')

1
2
3
4
5
6
7
8

response = kinetica.create_proc(
    proc_name = PROC_NAME,
    execution_mode = "distributed",
    files = file_map,
    command = "python",
    args = [PROC_FILE_NAME],
    options = {"set_environment": PROC_ENV_NAME}
)

1
2

ALTER FUNCTION ENVIRONMENT udfe_st
UNINSTALL PACKAGE 'sentence-transformers urllib3'

1
2
3
4
5
6
7

PROC_ENV_NAME = 'udfe_st'

kinetica.alter_environment(
    environment_name = PROC_ENV_NAME,
    action = "uninstall_package",
    value = "urllib3 sentence-transformers"
)

1

DROP FUNCTION ENVIRONMENT udfe_st

1

kinetica.drop_environment(environment_name = PROC_ENV_NAME, options = {"no_error_if_not_exists": "true"})

Initializing

A UDF must first get a handle to the ProcData class imported. This will parse the primary control file and set up all the necessary structures. It will return a ProcData instance, which is used to access data. All configuration information is cached, so repeated calls to get a handle to ProcData will not reload any configuration files.

Column Types

Unlike the other Kinetica APIs, the Python UDF API does not process data using records or schemas, operating in terms of columns of data instead. The raw column values returned closely map to the data types used in the tables being accessed:

Column data values can be accessed through array indices:

column[i]

For example, to retrieve the value for the 10th record:

column[9]

Reading Input Data

Accessing the request information for the UDF, the parameters passed into the UDF, or the input data can be completed using the following calls:

proc_data.request_info[<map_key>]

proc_data.input_data["customer"]["name"]
proc_data.input_data[5][1]

Writing Output Data

To output data to a table, the size of the table must be set in order to allocate enough space in all of the columns to hold the correct number of values. To do this, call:

table.size = <total number of output records>

Table column values can then be assigned to each OutputColumn of each OutputTable in the OutputDataSet.

The following calls are available to assist with writing data to Kinetica:

proc_data.output_data["customer"]["name"]
proc_data.output_data[5][1]

proc_data.output_data["customer"][1][2] = 12.34

proc_data.output_data[5]["name"][4].append("Joe")

Status Reporting

The proc_data.status property can be set to a string value to help convey status information during UDF execution, e.g., proc_data.status="25% complete". The show_proc_status() function will return any status messages set for each data segment -- one data segment per processing node if in distributed mode, or one data segment total in non-distributed mode. These messages are subject to the following scenarios:

• If the user-provided UDF code is executing and has set a status message, show_proc_status() will return the last message that was set.

• If the user-provided UDF code finishes executing successfully, the status message is cleared.

  Note

  The UDF may not show as "complete" yet since any data written by the UDF (in distributed mode) still has to be inserted into the database, but the status set by the UDF code isn't relevant to this process

• If the UDF is killed while executing user-provided UDF code, show_proc_status() will return the last message that was set.

• If the user-provided UDF code errors out, show_proc_status() will return the error message and the last status message that was set in parentheses.

Complete

The UDF must finish with a call to proc_data.complete. This writes out some final control information to indicate that the UDF completed successfully.

Logging

Any output from the UDF to is written to AWS CloudWatch.