Multi-Head Ingest

Multi-Head Ingest is a mechanism that allows sharded data to be ingested directly into cluster nodes, bypassing the overhead of pushing the data through the head node to the cluster nodes.

Operationally, the ingest mechanism calculates the target shard of each record to insert, and sends batches of would-be co-located records to their respective target nodes.

To fully utilize multi-head ingest, the ingestion process should use a multi-node parallel processing framework, such as MapReduce, Storm, or Spark. As data is divided up and flows through the processing nodes, a Kinetica bulk ingest object can be instantiated in each node to push data to the proper Kinetica cluster node. This greatly increases the speed of ingestion by both parallelizing the ingestion process and spreading the network traffic across multiple nodes.

API Support

Language Multi-head Ingest Mechanism
C++ GPUdbIngestor
C# KineticaIngestor
Java BulkInserter
Javascript X
Node.js X
Python GPUdbIngestor
SQL <automatic>


Kinetica in Azure automatically configures the database for multi-head ingest.

Clients which use the native APIs will also automatically be configured for multi-head ingest upon connecting to the database.


There are several factors to consider when using multi-head ingest:

  • Replicated tables are not supported
  • There is a small performance penalty for calculating the shard key of each record to be inserted into a sharded table.
  • There is an additional per-record performance penalty for primary key collision checks for any target table that has a primary key that is not the same as its shard key (the shard key columns are a proper subset of the primary key columns)
  • Randomly-sharded tables benefit more from multi-head ingest than sharded tables do, as there is no target shard key to calculate for each record and, therefore, no associated performance penalty for that calculation.
  • The batch size used to configure the bulk ingest object determines the record threshold for each of the insert queues targeting the worker nodes, not the record threshold for the bulk ingest object itself. Thus, ingestion into tables with non-uniform record distribution may require periodic flushes of the bulk ingest queues to ensure timely insertion of records in queues that infrequently reach their threshold.


All the functionality for multi-head ingestion is encapsulated in the bulk ingest object. See API Support for chart listing the API-specific object to use for bulk ingestion.

The following is a Java API code block that demonstrates the use of the BulkInserter for ingesting data.

// Create a bulk inserter for batch data ingestion
BulkInserter<MyType> bulkInserter = new BulkInserter<MyType>(gpudb, tableName, type, batchSize, null, new WorkerList(gpudb));

// Generate data to be inserted into the table, automatically inserting
// records whenever the batchSize limit is reached
for (int i = 0; i < numRecords; i++)
  MyType record = new MyType();
  record.put( 0, (i + 0.1) ); // col1
  record.put( 1, ("string " + String.valueOf( i ) ) ); // col2
  record.put( 2, "Group 1" );  // group_id
  bulkInserter.insert( record );

// To ensure all records are inserted, flush the bulk inserter object.

Python Background Multihead

In the Python API, the GPUdbTable object has a built-in use_multihead_io parameter, which allows the GPUdbTable to handle all Ingestor interactions with the associated table in the background:

ingest_table = gpudb.GPUdbTable(_type=ingest_columns, name="ingest_table", db=h_db, use_multihead_io=True)


If using the Java API and MapReduce, there is a conflict between the version of Avro used by Kinetica and the one used by MapReduce. This conflict can be overcome by using the Maven shade plug-in with the relocation tag: