Union

Kinetica supports the concept of set union, which is a representation of all rows that appear across a group of specified data sets (tables or views). Unions on collections are not supported.

A union is performed using the /create/union endpoint, using one of the following modes:

• Union -- retains all unique rows from the specified data sets
• Union Distinct -- synonymous with Union
• Union All -- retains all rows from the specified data sets
• Merge Views -- merge two or more filtered views (or filtered views of filtered views) of the same base data set into a new view. The resulting view would match the results of a SQL OR operation, e.g., if filter 1 creates a filtered view using the expression x = 10 and filter 2 creates a filtered view using the expression x <= 10, then the merge views operation creates a new view using the expression x = 10 OR x <= 10

Note

Set intersection and set substraction are also available, but their descriptions and limitations can be found on Intersect and Except, respectively.

You can union any number or combination of data sets as long as the columns across the data sets being used have similar data types. Kinetica will cast compatible data types as follows:

Type	Compatible With	Result Type
int8	int8	int8
int16	int16	int16
int	int	int
	long	long
	float	float
	double	double
	decimal	decimal
long	int	long
	long
	float	float
	double	double
	decimal	decimal
float	int	float
	long
	float
	double	double
	decimal	float
double	int	double
	long
	float
	double
	decimal
decimal	int	decimal
	long
	float	float
	double	double
	decimal	decimal
date	date	date
	datetime	datetime
	timestamp	timestamp
time	time	time
datetime	date	datetime
	datetime
timestamp	date	timestamp
	timestamp
charN	charN	charN (whichever N is larger)
wkt	wkt	wkt
ipv4	ipv4	ipv4

Performing a union creates a separate in-memory table containing the results. Unions can be persisted (like tables) using the persist option.

Note that if the source data sets are replicated, the results of the union will also be replicated. If the included data sets are sharded, the resulting in-memory table from the union will also be sharded; this also means that if a non-sharded data set is included, the resulting in-memory table will also be non-sharded.

Limitations on using union are discussed in further detail in the Limitations and Cautions section.

Performing a Union

To perform a union of data sets, the /create/union endpoint requires five parameters:

1. the name of the in-memory table to be created
2. the list of member data sets to be used in the union operation
3. the list of columns from each of the given data sets to be used in the union operation
4. the list of column names to be output to the resulting in-memory table
5. the union mode specified in the options input parameter
   • union_all (the default option)
   • union or union_distinct
   • merge_views -- Note that if this option is selected, the input_column_names AND output_column_names parameters are ignored

Note

If you do not specify a union mode, union_all will be used.

Examples

In Python, a union all between tables lunch_menu and dinner_menu would look like:

gpudb.create_union(
  table_name = "lunch_union_dinner",
  table_names = ["lunch_menu", "dinner_menu"],
  input_column_names = [
    ["food_name", "category", "price"],
    ["food_name", "category", "price"]
  ],
  output_column_names = ["food_name", "category", "price"]
)

The results from the above call would contain all menu items (including duplicates) in the extracted columns from lunch_menu and dinner_menu. The result would match what would be produced by the SQL:

SELECT
  food_name,
  category,
  price
FROM
  lunch_menu

UNION ALL

SELECT
  food_name,
  category,
  price
FROM
  dinner_menu

A union (or union distinct) using the same tables can be performed via:

gpudb.create_union(
  table_name = "lunch_union_dinner",
  table_names = ["lunch_menu", "dinner_menu"],
  input_column_names = [
    ["food_name", "category", "price"],
    ["food_name", "category", "price"]
  ],
  output_column_names = ["food_name", "category", "price"],
  options = {"mode":"union_distinct"}
)

The result of the union distinct call above would contain all menu items (excluding duplicates) in the extracted columns from lunch_menu and dinner_menu. The result would match what would be produced by the SQL:

SELECT
  food_name,
  category,
  price
FROM
  lunch_menu

UNION

SELECT
  food_name,
  category,
  price
FROM
  dinner_menu

A merge view can be performed via:

gpudb.create_union(
  table_name = "sandwiches_merged_view",
  table_names = ["lunch_sandwiches", "dinner_sandwiches"],
  options = {"mode":"merge_views"}
)

Operating on a Union

Examples

A Python example filter on the results produced in the Performing a Union section for menu items that are soups is:

gpudb.filter(
    table_name = "lunch_union_dinner",
    view_name = "menu_soups",
    expression = "category = soup"
)

When executed against a union, the /filter endpoint produces a filtered view. A chain of these filters could be used to create more and more restrictive views from the original union operation.

In Python, to aggregate menu items by category in the results produced earlier:

gpudb.aggregate_group_by(
  table_name = "lunch_union_dinner",
  column_names = [
    "soup"
  ],
  offset = 0,
  limit = 25,
  encoding = "json"
)

To retrieve records from the union results in Python:

gpudb.get_records(
    table_name = "lunch_union_dinner",
    offset = 0,
    limit = 50,
    encoding = "json"
)

Limitations and Cautions

There are several limitations to creating and using unions:

• Performing a union between two data sets results in an entirely new data set, so be mindful of the memory usage implications
• All data sets have to be replicated or not replicated, e.g., you cannot union replicated and non-replicated data sets
• If attempting to perform a distinct union on sharded data sets, all data sets have to be sharded similarly (if all data is not on the same processing node, the distinct union can't be calculated properly)
• The result of a union operation does not get updated if source data set(s) are updated
• A union has no permanence by default because it is not protected and will expire after the default TTL setting
• The input_column_name parameter vector size needs to match the number of data sets listed, i.e. if you want to union a data set to itself, the data set will need to be listed twice in the table_names parameter
• The input_column_name parameter vectors need to be listed in the same order as their source data sets, e.g., if three data sets are listed in the table_names parameter, the first data set's columns should be listed first in the input_column_name parameter, etc.
• Store-only columns and non-charN string column types cannot have union operations applied to them
• A merge views operation must consist of filtered views from the same base data set. This means filtered views of filtered views of the same base data set are allowed, but join views are not