com.gpudb

Class BulkInserter<T>

java.lang.Object

com.gpudb.BulkInserter<T>

Type Parameters:

T - the type of object being inserted

All Implemented Interfaces:

AutoCloseable

public class BulkInserter<T>
extends Object
implements AutoCloseable

Object that manages the insertion into GPUdb of large numbers of records in bulk, with automatic batch management and support for multi-head ingest. BulkInserter instances are thread safe and may be used from any number of threads simultaneously. Use the insert(Object) and insert(List) methods to queue records for insertion, and the flush() method to ensure that all queued records have been inserted.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BulkInserter.FlushOptions This class facilitates customizing the behavior of automatic flush in BulkInserter The default value of the 'flushInterval' is set to a negative value to indicate that the automatic flush feature is not needed.
static class	BulkInserter.InsertException An exception that occurred during the insertion of records into GPUdb.
static class	BulkInserter.WorkerList Deprecated. This class has been superseded by com.gpudb.WorkerList.

Constructor Summary

Constructors

Constructor and Description
BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions, GPUdbBase.JsonOptions jsonOptions) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers, GPUdbBase.JsonOptions jsonOptions) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions, GPUdbBase.JsonOptions jsonOptions) Creates a BulkInserter with the specified parameters.
BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers, GPUdbBase.JsonOptions jsonOptions) Creates a BulkInserter with the specified parameters.

Method Summary

Modifier and Type	Method and Description
void	close() This method will be called automatically if the BulkInserter class is used in a try-with-resources block.
void	flush() Ensures that any queued records are inserted into GPUdb.
int	getBatchSize() Gets the batch size (the number of records to insert into GPUdb at a time).
long	getCountInserted() Gets the number of records inserted into GPUdb.
long	getCountUpdated() Gets the number of records updated (instead of inserted) in GPUdb due to primary key conflicts.
List<BulkInserter.InsertException>	getErrors() Gets the list of errors received since the last call to getErrors().
GPUdb	getGPUdb() Gets the GPUdb instance into which records will be inserted.
GPUdbBase.JsonOptions	getJsonOptions()
int	getMaxRetries() Gets the number of times inserts into GPUdb will be retried in the event of an error.
Map<String,String>	getOptions() Gets the optional parameters that will be passed to GPUdb while inserting.
int	getRetryCount() Deprecated.
String	getTableName() Gets the name of the table into which records will be inserted.
BulkInserter.FlushOptions	getTimedFlushOptions() Returns the BulkInserter.FlushOptions instance value, useful for debugging.
List<BulkInserter.InsertException>	getWarnings() Gets the list of warnings received since the last call to getWarnings().
void	insert(List<T> records) Queues a list of records for insertion into GPUdb.
void	insert(T record) Queues a record for insertion into GPUdb.
boolean	isMultiHeadEnabled()
void	setJsonOptions(GPUdbBase.JsonOptions jsonOptions) This method could be used to set GPUdbBase.JsonOptions in case the user wants to modify the defaults.
void	setMaxRetries(int value) Sets the number of times inserts into GPUdb will be retried in the event of an error.
void	setRetryCount(int value) Deprecated.
void	setTimedFlushOptions(BulkInserter.FlushOptions flushOptions) This method could potentially result in two different scenarios 1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    Type type,
                    int batchSize,
                    Map<String,String> options)
             throws GPUdbException

Creates a BulkInserter with the specified parameters. This constructor could be used to obtain an instance of the BulkInserter class when the API user chooses not to use the multi-head ingestion facility even if it is turned on, on the server.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - the table to insert records into

type - the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached)

options - optional parameters to pass to GPUdb while inserting
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    Type type,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

type - the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    Type type,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers,
                    BulkInserter.FlushOptions flushOptions)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

type - the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

flushOptions - - instance of BulkInserter.FlushOptions class

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options, BulkInserter.FlushOptions

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    Type type,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers,
                    GPUdbBase.JsonOptions jsonOptions)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

type - the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

jsonOptions - - instance of GPUdbBase.JsonOptions class

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options, BulkInserter.FlushOptions, GPUdbBase.JsonOptions

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    Type type,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers,
                    BulkInserter.FlushOptions flushOptions,
                    GPUdbBase.JsonOptions jsonOptions)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

type - the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

flushOptions - - instance of BulkInserter.FlushOptions class

jsonOptions - - instance of GPUdbBase.JsonOptions class

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options, BulkInserter.FlushOptions, GPUdbBase.JsonOptions

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    TypeObjectMap<T> typeObjectMap,
                    int batchSize,
                    Map<String,String> options)
             throws GPUdbException

Creates a BulkInserter with the specified parameters. This constructor could be used to obtain an instance of the BulkInserter class when the API user chooses not to use the multi-head ingestion facility even if it is turned on, on the server.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

typeObjectMap - type object map for the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached)

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    TypeObjectMap<T> typeObjectMap,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

typeObjectMap - type object map for the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    TypeObjectMap<T> typeObjectMap,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers,
                    BulkInserter.FlushOptions flushOptions)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

typeObjectMap - type object map for the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

flushOptions - - instance of timed flush options BulkInserter.FlushOptions

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options, BulkInserter.FlushOptions, GPUdbBase.JsonOptions

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    TypeObjectMap<T> typeObjectMap,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers,
                    GPUdbBase.JsonOptions jsonOptions)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

typeObjectMap - type object map for the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

jsonOptions - - instance of GPUdbBase.JsonOptions class

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options, BulkInserter.FlushOptions, GPUdbBase.JsonOptions

BulkInserter

public BulkInserter(GPUdb gpudb,
                    String tableName,
                    TypeObjectMap<T> typeObjectMap,
                    int batchSize,
                    Map<String,String> options,
                    WorkerList workers,
                    BulkInserter.FlushOptions flushOptions,
                    GPUdbBase.JsonOptions jsonOptions)
             throws GPUdbException

Creates a BulkInserter with the specified parameters.

Parameters:

gpudb - the GPUdb instance to insert records into

tableName - name of the table to insert records into

typeObjectMap - type object map for the type of records being inserted

batchSize - the number of records to insert into GPUdb at a time (records will queue until this number is reached); for multi-head ingest, this value is per worker

options - optional parameters to pass to GPUdb while inserting (null for no parameters)
This is the same set of options as accepted by the GPUdb.insertRecords(String, List, Map) call.
The details can be found at InsertRecordsRequest.Options.

workers - worker list for multi-head ingest (null to disable multi-head ingest)

flushOptions - - instance of timed flush options BulkInserter.FlushOptions

jsonOptions - - instance of GPUdbBase.JsonOptions class

Throws:

GPUdbException - if a configuration error occurs

IllegalArgumentException - if an invalid parameter is specified

See Also:

InsertRecordsRequest.Options, BulkInserter.FlushOptions, GPUdbBase.JsonOptions

Method Detail

getTimedFlushOptions

public BulkInserter.FlushOptions getTimedFlushOptions()

Returns the BulkInserter.FlushOptions instance value, useful for debugging.

Returns:

- the BulkInserter.FlushOptions instance

getJsonOptions

public GPUdbBase.JsonOptions getJsonOptions()

setTimedFlushOptions

public void setTimedFlushOptions(BulkInserter.FlushOptions flushOptions)
                          throws GPUdbException

This method could potentially result in two different scenarios 1. It could start a timed flush thread if it was not already active when the BulkInserter was created. 2. If the timed flush thread was already active then setting this to a new value will first terminate the existing thread; set the options to the new value and finally restart a new thread with the options passed in. This case could result in a delay since the thread needs to be cleaned up and restarted.

Parameters:

flushOptions - - an instance of the BulkInserter.FlushOptions class

Throws:

GPUdbException - - in case an invalid timeout values is set in the flushOptions parameter.

setJsonOptions

public void setJsonOptions(GPUdbBase.JsonOptions jsonOptions)

This method could be used to set GPUdbBase.JsonOptions in case the user wants to modify the defaults.

Parameters:

jsonOptions -

See Also:

GPUdbBase.JsonOptions

close

public void close()
           throws BulkInserter.InsertException

This method will be called automatically if the BulkInserter class is used in a try-with-resources block. If not used that way it is mandatory to call this method to initiate a smooth cleanup of the underlying resources. This method will terminate the internal scheduler threads and call flush so that all pending updates to the database are handled properly.

     try( BulkInserter inserter = new BulkInserter(...) ) {
         // Do something with the BulkInserter instance
         // inserter.{some_method}
     }
     // Here the close method of the BulkInserter class will be called
     // automatically
 

or

     BulkInserter inserter = new BulkInserter<>(...)
     // Invoke some methods on the inserter
     //Explicitly call close() method
     inserter.close();
 

Specified by:

close in interface AutoCloseable

Throws:

BulkInserter.InsertException - - While doing the final flush

getGPUdb

public GPUdb getGPUdb()

Gets the GPUdb instance into which records will be inserted.

Returns:

the GPUdb instance into which records will be inserted

getTableName

public String getTableName()

Gets the name of the table into which records will be inserted.

Returns:

the name of the table into which records will be inserted

getBatchSize

public int getBatchSize()

Gets the batch size (the number of records to insert into GPUdb at a time). For multi-head ingest this value is per worker.

Returns:

the batch size

getOptions

public Map<String,String> getOptions()

Gets the optional parameters that will be passed to GPUdb while inserting.

Returns:

the optional parameters that will be passed to GPUdb while inserting

See Also:

InsertRecordsRequest.Options

isMultiHeadEnabled

public boolean isMultiHeadEnabled()
                           throws GPUdbException

Throws:

GPUdbException

getMaxRetries

public int getMaxRetries()

Gets the number of times inserts into GPUdb will be retried in the event of an error. After this many retries, BulkInserter.InsertException will be thrown.

Returns:

the number of retries

See Also:

setMaxRetries(int)

setMaxRetries

public void setMaxRetries(int value)

Sets the number of times inserts into GPUdb will be retried in the event of an error. After this many retries, BulkInserter.InsertException will be thrown.

Parameters:

value - the number of retries

Throws:

IllegalArgumentException - if value is less than zero

See Also:

getMaxRetries()

getRetryCount

@Deprecated
public int getRetryCount()

Deprecated.

Gets the number of times inserts into GPUdb will be retried in the event of an error. After this many retries, BulkInserter.InsertException will be thrown.

Returns:

the number of retries

See Also:

setRetryCount(int)

setRetryCount

@Deprecated
public void setRetryCount(int value)

Deprecated.

Sets the number of times inserts into GPUdb will be retried in the event of an error. After this many retries, BulkInserter.InsertException will be thrown.

Parameters:

value - the number of retries

Throws:

IllegalArgumentException - if value is less than zero

See Also:

getRetryCount()

getCountInserted

public long getCountInserted()

Gets the number of records inserted into GPUdb. Excludes records that are currently queued but not yet inserted and records not inserted due to primary key conflicts.

Returns:

the number of records inserted

getCountUpdated

public long getCountUpdated()

Gets the number of records updated (instead of inserted) in GPUdb due to primary key conflicts.

Returns:

the number of records updated

getErrors

public List<BulkInserter.InsertException> getErrors()

Gets the list of errors received since the last call to getErrors().

Returns:

list of InsertException objects

getWarnings

public List<BulkInserter.InsertException> getWarnings()

Gets the list of warnings received since the last call to getWarnings().

Returns:

list of InsertException objects

flush

public void flush()
           throws BulkInserter.InsertException

Ensures that any queued records are inserted into GPUdb. If an error occurs while inserting the records from any queue, the records will no longer be in that queue nor in GPUdb; catch BulkInserter.InsertException to get the list of records that were being inserted if needed (for example, to retry). Other queues may also still contain unflushed records if this occurs.

Throws:

BulkInserter.InsertException - if an error occurs while inserting

insert

public void insert(T record)
            throws BulkInserter.InsertException

Queues a record for insertion into GPUdb. If the queue reaches the batch size, all records in the queue will be inserted into GPUdb before the method returns. If an error occurs while inserting the records, the records will no longer be in the queue nor in GPUdb; catch BulkInserter.InsertException to get the list of records that were being inserted if needed (for example, to retry). Note: This version of insert(Object) will result in sequential batch inserts in a background thread. Use insert(List) to allow multiple queues to reach their batch size and parallelize all of those batch inserts at once.

Parameters:

record - the record to insert

Throws:

GPUdbException - if an error occurs while calculating shard/primary keys

BulkInserter.InsertException - if an error occurs while inserting

insert

public void insert(List<T> records)
            throws BulkInserter.InsertException

Queues a list of records for insertion into GPUdb. If any queue reaches the batch size, all queues that have reached the batch size will have their records inserted into the database before the method returns. If an error occurs while inserting the records, they will no longer be in that queue or the database; catch BulkInserter.InsertException to get the list of records that were being inserted (including any from the queue in question and any remaining in the list not yet queued) if needed (for example, to retry). Note that depending on the number of records, multiple calls to the database may occur. Note: This version of insert(List) will result in parallelizing the batch inserts in background threads.

Parameters:

records - the records to insert

Throws:

BulkInserter.InsertException - if an error occurs while inserting