Data

Ingests

An ingest is a means for getting data into Kinetica to be used in models.

  • Point to + Add New Ingest then click one of the options to begin creating an ingest.
  • If an ingest has an output table, click the output table name to view a preview of the table
  • Type into Filter to filter down the ingests
  • Click show to display any archived ingests
  • Click refresh to refresh the table
  • Click export to export the table's values as JSON or CSV
  • Click an existing ingest to display additional actions:
    • Click View Selection to open the Ingest Details page
    • Click Start to start the ingest job
    • Click Export Entity to export the ingest as a JSON object
    • Click Terminate to stop the ingest
    • Click Clone to clone the ingest
    • Click Archive to archive the ingest; it will be stopped (if not already) and hidden from the list of ingests
    • Click Description / Config to review summary information for the ingest
../img/aaw_ui_ingests.png

Details

The Ingest Details page provides a detailed look at a given ingest, including configuration information, state, and a data preview (if the ingest has a destination table configured).

  • Click Back to return to the Ingests page
  • Click Start to start the ingest job (if it's not running currently)
  • Click Export Entity to export the ingest as a JSON object
  • Click Terminate to remove the ingest
  • Click Clone to clone the ingest
  • Click Archive to archive the ingest; it will be stopped (if not already) and hidden from the list of ingests
../img/aaw_ui_ingest_details.png

Batch

A batch ingest uses an internal mechanism to ingest a set of data at once.

To create a batch ingest:

  1. Provide a name for the ingest

  2. Optionally, provide a description for the ingest, then click Next.

  3. Select a Source Type.

  4. Fill the rest of the Source and Destination fields, then click Next.

    Important

    All sources require a credential; consult Security for more information.

  5. Review the summary, then click Create.

Streaming

A streaming ingest uses Kafka to continuously stream data into Kinetica for use in AAW.

To create a streaming ingest:

  1. Provide a name for the ingest

  2. Optionally, provide a description for the ingest, then click Next.

  3. Fill the Source and Destination fields, then click Next.

    Important

    All sources require a credential; consult Security for more information.

  4. Review the summary, then click Create.

Import BYOC

A BYOC (bring your own container) ingest uses a Docker container to continuously ingest data into Kinetica for use in AAW.

To create a BYOC ingest:

  1. Provide a Docker container source URI, e.g.,

    <repo-name>/<image-name>:<tag-name>

    Tip

    Optionally, click Inspect to preview container metadata and pre-populate some fields.

  2. Provide a name for the ingest

  3. Optionally, provide a description for the ingest

  4. Add environment variable key value pairs:

    1. Click Add to create as many environment variables as necessary
    2. Provide a key name for each environment variable
    3. Provide a value for each environment variable

  5. Provide a destination table name for the ingest

  6. Click Import

Datasets

Datasets are used for providing the training and test data for models.

  • Click + Add New Dataset to begin the dataset setup process.
  • Click the Source Table name to display the table data
  • Type into Filter to filter down the datasets
  • Click show to display any archived datasets.
  • Click refresh to refresh the table
  • Click export to export the table's values as JSON or CSV
  • Click an existing dataset to display additional actions:
    • Click View Selection to open the Dataset Details page
    • Click Preview Data to open a dataset preview
    • Click Description / Config to review summary information for the dataset
    • Click Clone to clone the dataset
    • Click Export Entity to export the dataset as a JSON object
    • Click Archive to archive the dataset; it will be hidden from the list of datasets
  • Click Description / Config to review summary information for the dataset
../img/aaw_ui_datasets.png

Details

The Dataset Details page provides a detailed look at a given dataset, including configuration information and state.

  • In the Configuration table, click View to view the columns for the dataset
  • Click Back to return to the Datasets page
  • Click Preview Data to open a dataset preview
  • Click Clone to clone the dataset
  • Click Export Entity to export the dataset as a JSON object
  • Click Archive to archive the dataset; it will be hidden from the list of datasets
../img/aaw_ui_dataset_details.png

New Dataset

To create a new dataset:

  1. Provide a Name for the dataset.

  2. Optionally, provide a Description.

  3. Select a Source Table.

    Note

    The list of source tables is populated with tables in the Kinetica installation associated with this instance of AAW.

  4. Select one or more Columns from the table, or click Select All to select all columns.

  5. Optionally, provide a filter expression for the columns.

  6. Click Create.

Feature Sets

Feature Sets transform columns (features) from datasets using inline and relational transforms.

  • Click + Add New Feature Set to begin the feature set setup process.
  • Click the Dataset name to display the Dataset Details for the selected dataset
  • Type into Filter to filter down the feature sets
  • Click show to display any archived feature sets.
  • Click refresh to refresh the table
  • Click export to export the table's values as JSON or CSV
  • Click an existing feature set to display additional actions:
    • Click View Selection to open the Feature Set Details page
    • Click Clone to clone the feature set
    • Click Export Entity to export the feature set as a JSON object
    • Click Archive to archive the feature set; it will be hidden from the list of feature set
../img/aaw_ui_featuresets.png

Details

The Feature Set Details page provides a detailed look at a given feature set, including state, features, function, and lambda function information.

  • Click Back to return to the Feature Sets page
  • Click Clone to clone the feature set
  • Click Export Entity to export the feature set as a JSON object
  • Click Archive to archive the feature set; it will be hidden from the list of feature set
../img/aaw_ui_featureset_details.png

New Feature Set

To create a new feature set:

  1. Provide a Name for the feature set.

  2. Optionally, provide a Description.

  3. Select a transform type:

    • Inline Transforms -- Transforms static data using provided functions to prepare the data for use in models for all deployment types

      1. Select an existing Dataset.

      2. Create new features using one of the following methods:

        • Click Transform Feature to create new features one at a time.

          1. Provide a Feature Name.
          2. Select a Column from the selected dataset.
          3. Select one or more Function(s) to transform the column.

        • Click the Advanced Input slider and paste a JSON-formatted feature list into the text field.

          Tip

          When first enabling Advanced Input, an example input is displayed to help get you started on writing a JSON-formatted feature list

    • Relational Transform -- Allows for constantly changing data via a materialized view to be used in models for continuous deployments

      1. Provide a name for existing materialized view.
      2. Provide the name of a timestamp column in the materialized view.

  4. Click Create.