Running Java UDFs

The information below includes all information one needs to know to begin running Java UDFs. For more information on writing Java UDFs, see Java UDF API; for more information on simulating running UDFs, see UDF Simulator. Example Java UDFs can be found here.

Important

Though any of the native APIs can be used for running UDFs written in any UDF API language, all the examples below are written using the native Java API.

Deployment

Calling the createProc() method will deploy the specified UDF to the Kinetica execution environment, to every server in the cluster. The method takes the following parameters:

Parameter Description
procName A system-wide unique name for the UDF
executionMode An execution mode; either distributed or nondistributed
files

A set of files composing the UDF package, including the names of the files and the binary data for those files. The files specified will be created on the target Kinetica servers in the UDF directory, with the given data and filenames; if the filenames contain subdirectories, that structure will be copied to the target servers.

Files in KiFS can also be used here. To use a KiFS file, pass the KiFS URI as the name of the file and leave the binary data portion empty. In this case, the KiFS files will be copied into the UDF directory for use when executeProc() is called.

Note

Uploading files using the files parameter should be reserved for smaller files; larger files should be uploaded to KiFS and referenced there instead.

command The name of the command to run, which can be a file within the deployed UDF fileset, or any command able to be executed within the host environment, e.g., java. If a host environment command is specified, the host environment must be properly configured to support that command's execution.
args A list of command-line arguments to pass to the specified command; e.g., -cp <class path> <class name> or -jar <file name.jar>
options Optional parameters for UDF creation; see createProc() for details

For example, to deploy a Java UDF using the native Java API, a local, compiled proc jar (kinetica-udf-table-copy-proc-7.1.2.jar) will need to be read in as bytes and then passed into the createProc() call as a value paired with the file name as its key inside map files. A Java class path argument, which includes the proc jar and a list of server-side Java UDF APIs, is also provided. Lastly, the Java executable class is given.

Create UDF Example - File & Class Name Constants
1
2
3
4
5
6
7
8
9
static String CSV_FILE = "rank_tom.csv";
static String PROC_NAME = "UdfTcJavaProc";
static String PROC_PATH = "com.kinetica." + PROC_NAME;
static String PROC_JAR_FILE = "kinetica-udf-table-copy-proc-7.1.2.jar";
static String PROC_API_ROOT = "/opt/gpudb/udf/api/java/proc-api/";
static String PROC_API_7100_FILE = PROC_API_ROOT + "kinetica-proc-api-7.1.0.0-jar-with-dependencies.jar";
static String PROC_API_7101_FILE = PROC_API_ROOT + "kinetica-proc-api-7.1.0.1-jar-with-dependencies.jar";
static String PROC_API_7102_FILE = PROC_API_ROOT + "kinetica-proc-api-7.1.0.2-jar-with-dependencies.jar";
static String PROC_API_FILE = PROC_API_ROOT + "kinetica-proc-api.jar";
Create UDF Example - File Map Loading
1
2
3
4
5
6
7
Map<String, ByteBuffer> filesMap = new HashMap<>();
for (String fileName : Arrays.asList(CSV_FILE, PROC_JAR_FILE))
{
    byte [] fileAsBytes = Files.readAllBytes(new File(fileName).toPath());
    ByteBuffer fileByteBuffer = ByteBuffer.wrap(fileAsBytes);
    filesMap.put(fileName, fileByteBuffer);
}
Create UDF Example - createProc() Call
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
CreateProcResponse createProcResponse = kinetica.createProc(
        PROC_NAME,
        "distributed",
        filesMap,
        "java",
        Arrays.asList(
                "-cp",
                PROC_JAR_FILE + ":" +
                    PROC_API_7100_FILE + ":" +
                    PROC_API_7101_FILE + ":" +
                    PROC_API_7102_FILE + ":" +
                    PROC_API_FILE,
                PROC_PATH
        ),
        null
);

To use the same files, located in a KiFS directory named udf instead, initialize files as follows:

Create UDF Example - KiFS Filename Constants & File Map Loading
1
2
3
4
5
6
static String CSV_FILE = "kifs://udf/rank_tom.csv";
static String PROC_JAR_FILE = "kifs://udf/kinetica-udf-table-copy-proc-7.1.2.jar";

Map<String, ByteBuffer> filesMap = new HashMap<>();
for (String fileName : Arrays.asList(CSV_FILE, PROC_JAR_FILE))
    filesMap.put(fileName, null);

Concurrency Limits

The max_concurrency_per_node setting is available in the options map of the /create/proc. This option allows you to define a per-Kinetica- host concurrency limit for a UDF, i.e. no more than n OS processes (UDF instances) in charge of evaluating the UDF will be permitted to execute concurrently on a single Kinetica host. You may want to set a concurrency limit if you have limited resources (like GPUs) and want to avoid the risks of continually exhausting your resources. This setting is particularly useful for distributed UDFs, but it will also work for non-distributed UDFs.

Note

You can also set concurrency limits on the Edit Proc screen in the UDF section of GAdmin

The default value for the setting is 0, which results in no limits. If you set the value to 4, only 4 instances of the UDF will be queued to execute the UDF. This holds true across all invocations of the proc; this means that even if /execute/proc is called eight times, only 4 processes will be running. Another instance will be queued as soon as one instance finishes processing. This process will repeat, only allowing 4 instances of the UDF to run at a time, until all instances have completed or the UDF is killed.

Execution

Calling the executeProc() method will execute the specified UDF within the targeted Kinetica execution environment. The method takes the following parameters:

Parameter Description
procName The system-wide unique name for the UDF
params Set of string-to-string key/value paired parameters to pass to the UDF
binParams Set of string-to-binary key/value paired parameters to pass to the UDF
inputTableNames Input data table names, to be processed by the UDF
inputColumnNames Mapping of input data table names to their respective column names, to be processed as input data by the UDF
outputTableNames Output data table names, where processed data is to be appended
options Optional parameters for UDF execution; see executeProc() for details

The call is asynchronous and will return immediately with a run_id, which is a string that can be used in subsequent checks of the execution status.

For example, to execute a UDF that's already been created (UdfTcJavaProc) using existing input (udf_tc_java_in_table) and output (udf_tc_java_out_table) tables:

Execute UDF Example - Parameter Constants
1
static String PROC_NAME = "UdfTcJavaProc";
Execute UDF Example - executeProc() Call
1
2
3
4
5
6
7
8
9
ExecuteProcResponse executeProcResponse = kinetica.executeProc(
        PROC_NAME,
        null,
        null,
        Collections.singletonList(inputTable),
        null,
        Collections.singletonList(outputTable),
        null
);

Management

UDFs can be managed using SQL, GAdmin, or through one of the native API calls below:

  • deleteProc() -- removes the given UDF definition from the system; needs to be called before createProc() when recreating a UDF
  • hasProc() -- returns whether the given UDF exists
  • killProc() -- terminates a running UDF (or UDFs)
  • showProc() -- returns the parameter values used in creating the UDF
  • showProcStatus() -- Returns whether the UDF (or UDFs) is still running, has completed, or has exited with an error, along with any processed results