Version:

Union

Kinetica supports the concept of set union, which is a representation of all rows that appear across a group of specified data sets (tables or views). Unions on collections are not supported.

A union is performed using the /create/union endpoint, using one of the following modes:

  • Union -- retains all unique rows from the specified data sets
  • Union Distinct -- synonymous with Union
  • Union All -- retains all rows from the specified data sets
  • Merge Views -- merge two or more filtered views (or filtered views of filtered views) of the same base data set into a new view. The resulting view would match the results of a SQL OR operation, e.g., if filter 1 creates a filtered view using the expression x = 10 and filter 2 creates a filtered view using the expression x <= 10, then the merge views operation creates a new view using the expression x = 10 OR x <= 10

Note

Set intersection and set substraction are also available, but their descriptions and limitations can be found on Intersect and Except, respectively.

You can union any number or combination of data sets as long as the columns across the data sets being used have similar data types. Kinetica will cast compatible data types as follows:

Type Compatible With Result Type
int8 int8 int8
int16 int16 int16
int int int
long long
float float
double double
decimal decimal
long int long
long
float float
double double
decimal decimal
float int float
long
float
double double
decimal float
double int double
long
float
double
decimal
decimal int decimal
long
float float
double double
decimal decimal
date date date
datetime datetime
timestamp timestamp
time time time
datetime date datetime
datetime
timestamp date timestamp
timestamp
charN charN charN (whichever N is larger)
wkt wkt wkt
ipv4 ipv4 ipv4

Performing a union creates a separate in-memory table containing the results. Unions can be persisted (like tables) using the persist option.

Note that if the source data sets are replicated, the results of the union will also be replicated. If the included data sets are sharded, the resulting in-memory table from the union will also be sharded; this also means that if a non-sharded data set is included, the resulting in-memory table will also be non-sharded.

Limitations on using union are discussed in further detail in the Limitations and Cautions section.

Performing a Union

To perform a union of data sets, the /create/union endpoint requires five parameters:

  1. the name of the in-memory table to be created
  2. the list of member data sets to be used in the union operation
  3. the list of columns from each of the given data sets to be used in the union operation
  4. the list of column names to be output to the resulting in-memory table
  5. the union mode specified in the options input parameter
    • union_all (the default option)
    • union or union_distinct
    • merge_views -- Note that if this option is selected, the input_column_names AND output_column_names parameters are ignored

Note

If you do not specify a union mode, union_all will be used.

Examples

In Python, a union all between tables lunch_menu and dinner_menu would look like:

gpudb.create_union(
  table_name = "lunch_union_dinner",
  table_names = ["lunch_menu", "dinner_menu"],
  input_column_names = [
    ["food_name", "category", "price"],
    ["food_name", "category", "price"]
  ],
  output_column_names = ["food_name", "category", "price"]
)

The results from the above call would contain all menu items (including duplicates) in the extracted columns from lunch_menu and dinner_menu. The result would match what would be produced by the SQL:

SELECT
  food_name,
  category,
  price
FROM
  lunch_menu

UNION ALL

SELECT
  food_name,
  category,
  price
FROM
  dinner_menu

A union (or union distinct) using the same tables can be performed via:

gpudb.create_union(
  table_name = "lunch_union_dinner",
  table_names = ["lunch_menu", "dinner_menu"],
  input_column_names = [
    ["food_name", "category", "price"],
    ["food_name", "category", "price"]
  ],
  output_column_names = ["food_name", "category", "price"],
  options = {"mode":"union_distinct"}
)

The result of the union distinct call above would contain all menu items (excluding duplicates) in the extracted columns from lunch_menu and dinner_menu. The result would match what would be produced by the SQL:

SELECT
  food_name,
  category,
  price
FROM
  lunch_menu

UNION

SELECT
  food_name,
  category,
  price
FROM
  dinner_menu

A merge view can be performed via:

gpudb.create_union(
  table_name = "sandwiches_merged_view",
  table_names = ["lunch_sandwiches", "dinner_sandwiches"],
  options = {"mode":"merge_views"}
)

Operating on a Union

Examples

A Python example filter on the results produced in the Performing a Union section for menu items that are soups is:

gpudb.filter(
    table_name = "lunch_union_dinner",
    view_name = "menu_soups",
    expression = "category = soup"
)

When executed against a union, the /filter endpoint produces a filtered view. A chain of these filters could be used to create more and more restrictive views from the original union operation.

In Python, to aggregate menu items by category in the results produced earlier:

gpudb.aggregate_group_by(
  table_name = "lunch_union_dinner",
  column_names = [
    "soup"
  ],
  offset = 0,
  limit = 25,
  encoding = "json"
)

To retrieve records from the union results in Python:

gpudb.get_records(
    table_name = "lunch_union_dinner",
    offset = 0,
    limit = 50,
    encoding = "json"
)

Limitations and Cautions

There are several limitations to creating and using unions:

  • Performing a union between two data sets results in an entirely new data set, so be mindful of the memory usage implications
  • All data sets have to be replicated or not replicated, e.g., you cannot union replicated and non-replicated data sets
  • If attempting to perform a distinct union on sharded data sets, all data sets have to be sharded similarly (if all data is not on the same processing node, the distinct union can't be calculated properly)
  • The result of a union operation does not get updated if source data set(s) are updated
  • A union has no permanence by default because it is not protected and will expire after the default TTL setting
  • The input_column_name parameter vector size needs to match the number of data sets listed, i.e. if you want to union a data set to itself, the data set will need to be listed twice in the table_names parameter
  • The input_column_name parameter vectors need to be listed in the same order as their source data sets, e.g., if three data sets are listed in the table_names parameter, the first data set's columns should be listed first in the input_column_name parameter, etc.
  • Store-only columns and non-charN string column types cannot have union operations applied to them
  • A merge views operation must consist of filtered views from the same base data set. This means filtered views of filtered views of the same base data set are allowed, but join views are not