T - the type of object being insertedpublic class BulkInserter<T> extends Object implements AutoCloseable
BulkInserter instances are thread safe and may be used from any
number of threads simultaneously. Use the insert(Object) and
insert(List) methods to queue records for insertion, and the
flush() method to ensure that all queued records have been inserted.| Modifier and Type | Class and Description |
|---|---|
static class |
BulkInserter.FlushOptions
This class facilitates customizing the behavior of automatic flush in
BulkInserter
The default value of the 'flushInterval' is set to a negative value to indicate that
the automatic flush feature is not needed. |
static class |
BulkInserter.InsertException
An exception that occurred during the insertion of records into GPUdb.
|
static class |
BulkInserter.WorkerList
Deprecated.
This class has been superseded by
com.gpudb.WorkerList. |
| Constructor and Description |
|---|
BulkInserter(GPUdb gpudb,
String tableName,
Type type,
int batchSize)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
Type type,
int batchSize,
Map<String,String> options)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
Type type,
int batchSize,
Map<String,String> options,
WorkerList workers)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
Type type,
int batchSize,
Map<String,String> options,
WorkerList workers,
BulkInserter.FlushOptions flushOptions)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
Type type,
int batchSize,
Map<String,String> options,
WorkerList workers,
BulkInserter.FlushOptions flushOptions,
GPUdbBase.JsonOptions jsonOptions)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
Type type,
int batchSize,
Map<String,String> options,
WorkerList workers,
GPUdbBase.JsonOptions jsonOptions)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
TypeObjectMap<T> typeObjectMap,
int batchSize)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
TypeObjectMap<T> typeObjectMap,
int batchSize,
Map<String,String> options)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
TypeObjectMap<T> typeObjectMap,
int batchSize,
Map<String,String> options,
WorkerList workers)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
TypeObjectMap<T> typeObjectMap,
int batchSize,
Map<String,String> options,
WorkerList workers,
BulkInserter.FlushOptions flushOptions)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
TypeObjectMap<T> typeObjectMap,
int batchSize,
Map<String,String> options,
WorkerList workers,
BulkInserter.FlushOptions flushOptions,
GPUdbBase.JsonOptions jsonOptions)
Creates a
BulkInserter with the specified parameters. |
BulkInserter(GPUdb gpudb,
String tableName,
TypeObjectMap<T> typeObjectMap,
int batchSize,
Map<String,String> options,
WorkerList workers,
GPUdbBase.JsonOptions jsonOptions)
Creates a
BulkInserter with the specified parameters. |
| Modifier and Type | Method and Description |
|---|---|
void |
close()
Closes
BulkInserter resources:
Performs a final flush of any data yet to be ingested
Terminates the timed flush mechanism, if needed
Terminates the ingest executor service
This method will be called automatically if the BulkInserter class is used in a
try-with-resources block. |
void |
flush()
Ensures that any queued records are inserted into GPUdb.
|
int |
getBatchSize()
Gets the batch size (the number of records to insert into GPUdb at a
time).
|
long |
getCountInserted()
Gets the number of records inserted into GPUdb.
|
long |
getCountUpdated()
Gets the number of records updated (instead of inserted) in GPUdb due to
primary key conflicts.
|
List<BulkInserter.InsertException> |
getErrors()
Gets the list of errors received since the last call to getErrors().
|
GPUdb |
getGPUdb()
Gets the GPUdb instance into which records will be inserted.
|
GPUdbBase.JsonOptions |
getJsonOptions()
Retrieves options governing JSON ingest.
|
int |
getMaxRetries()
Gets the number of times inserts into GPUdb will be retried in the event
of an error.
|
Map<String,String> |
getOptions()
Gets the optional parameters that will be passed to GPUdb while
inserting.
|
int |
getRetryCount()
Deprecated.
|
String |
getTableName()
Gets the name of the table into which records will be inserted.
|
BulkInserter.FlushOptions |
getTimedFlushOptions()
Retrieves options governing the timed flush ingest scheme.
|
List<BulkInserter.InsertException> |
getWarnings()
Gets the list of warnings received since the last call to getWarnings().
|
void |
insert(List<T> records)
Queues a list of records for insertion into GPUdb.
|
void |
insert(T record)
Queues a record for insertion into GPUdb.
|
boolean |
isMultiHeadEnabled() |
void |
setJsonOptions(GPUdbBase.JsonOptions jsonOptions)
This method could be used to set
GPUdbBase.JsonOptions in case the user wants to
modify the defaults. |
void |
setMaxRetries(int value)
Sets the number of times inserts into GPUdb will be retried in the event
of an error.
|
void |
setRetryCount(int value)
Deprecated.
|
void |
setTimedFlushOptions(BulkInserter.FlushOptions flushOptions)
This method could potentially result in two different scenarios
It could start a timed flush thread if it was not already active
when the BulkInserter was created.
|
public BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize) throws GPUdbException
BulkInserter with the specified parameters.
GPUdb.insertRecords(String, List, Map) call. Details can be
found at InsertRecordsRequest.Options.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotype - the Type of records being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workerGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Optionspublic BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options) throws GPUdbException
BulkInserter with the specified parameters.
gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotype - the Type of records being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.GPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Optionspublic BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotype - the Type of records being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Optionspublic BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotype - the Type of records being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headflushOptions - BulkInserter.FlushOptions to use for timed flush operationGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Options,
BulkInserter.FlushOptionspublic BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers, GPUdbBase.JsonOptions jsonOptions) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotype - the Type of records being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headjsonOptions - GPUdbBase.JsonOptions to use for JSON ingestGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Options,
BulkInserter.FlushOptions,
GPUdbBase.JsonOptionspublic BulkInserter(GPUdb gpudb, String tableName, Type type, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions, GPUdbBase.JsonOptions jsonOptions) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotype - the Type of records being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headflushOptions - BulkInserter.FlushOptions to use for timed flush operationjsonOptions - GPUdbBase.JsonOptions to use for JSON ingestGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Options,
BulkInserter.FlushOptions,
GPUdbBase.JsonOptionspublic BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize) throws GPUdbException
BulkInserter with the specified parameters.
GPUdb.insertRecords(String, List, Map) call. Details can be
found at InsertRecordsRequest.Options.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotypeObjectMap - the TypeObjectMap for the type of records
being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workerGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Optionspublic BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options) throws GPUdbException
BulkInserter with the specified parameters.
gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotypeObjectMap - the TypeObjectMap for the type of records
being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.GPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Optionspublic BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotypeObjectMap - the TypeObjectMap for the type of records
being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Optionspublic BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotypeObjectMap - the TypeObjectMap for the type of records
being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headflushOptions - BulkInserter.FlushOptions to use for timed flush operationGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Options,
BulkInserter.FlushOptions,
GPUdbBase.JsonOptionspublic BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers, GPUdbBase.JsonOptions jsonOptions) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotypeObjectMap - the TypeObjectMap for the type of records
being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headjsonOptions - GPUdbBase.JsonOptions to use for JSON ingestGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Options,
BulkInserter.FlushOptions,
GPUdbBase.JsonOptionspublic BulkInserter(GPUdb gpudb, String tableName, TypeObjectMap<T> typeObjectMap, int batchSize, Map<String,String> options, WorkerList workers, BulkInserter.FlushOptions flushOptions, GPUdbBase.JsonOptions jsonOptions) throws GPUdbException
BulkInserter with the specified parameters.gpudb - the GPUdb instance to insert records intotableName - name of the table to insert records intotypeObjectMap - the TypeObjectMap for the type of records
being insertedbatchSize - the number of records to insert into GPUdb at a time
(records will queue until this number is reached);
for multi-head ingest, this value is per workeroptions - optional parameters to pass to GPUdb while inserting
(null for no parameters)
GPUdb.insertRecords(String, List, Map) call.
InsertRecordsRequest.Options.workers - worker list for multi-head ingest; use an empty worker
list (new WorkerList()) to disable multi-headflushOptions - BulkInserter.FlushOptions to use for timed flush operationjsonOptions - GPUdbBase.JsonOptions to use for JSON ingestGPUdbException - if a configuration error occursIllegalArgumentException - if an invalid parameter is specifiedInsertRecordsRequest.Options,
BulkInserter.FlushOptions,
GPUdbBase.JsonOptionspublic BulkInserter.FlushOptions getTimedFlushOptions()
BulkInserter.FlushOptions instancepublic GPUdbBase.JsonOptions getJsonOptions()
GPUdbBase.JsonOptions instancepublic void setTimedFlushOptions(BulkInserter.FlushOptions flushOptions) throws GPUdbException
flushOptions - BulkInserter.FlushOptions to use for timed flush operationGPUdbException - in case an invalid timeout values is set in the
flushOptions parameter.public void setJsonOptions(GPUdbBase.JsonOptions jsonOptions)
GPUdbBase.JsonOptions in case the user wants to
modify the defaults.jsonOptions - GPUdbBase.JsonOptions to use for JSON ingestGPUdbBase.JsonOptionspublic void close()
throws BulkInserter.InsertException
BulkInserter resources:
BulkInserter class is used in a
try-with-resources block. If not used that way it is mandatory to call this method
to initiate a smooth cleanup of the underlying resources.
try( BulkInserter inserter = new BulkInserter(...) ) {
// Do something with the BulkInserter instance
// inserter.{some_method}
}
// Here the close method of the BulkInserter class will be called
// automatically
or
BulkInserter inserter = new BulkInserter<>(...)
// Invoke some methods on the inserter
//Explicitly call close() method
inserter.close();
close in interface AutoCloseableBulkInserter.InsertException - - While doing the final flushpublic GPUdb getGPUdb()
public String getTableName()
public int getBatchSize()
public Map<String,String> getOptions()
InsertRecordsRequest.Optionspublic boolean isMultiHeadEnabled()
public int getMaxRetries()
BulkInserter.InsertException will be
thrown.setMaxRetries(int)public void setMaxRetries(int value)
BulkInserter.InsertException will be
thrown.value - the number of retriesIllegalArgumentException - if value is less than zerogetMaxRetries()@Deprecated public int getRetryCount()
BulkInserter.InsertException will be
thrown.setRetryCount(int)@Deprecated public void setRetryCount(int value)
BulkInserter.InsertException will be
thrown.value - the number of retriesIllegalArgumentException - if value is less than zerogetRetryCount()public long getCountInserted()
public long getCountUpdated()
public List<BulkInserter.InsertException> getErrors()
public List<BulkInserter.InsertException> getWarnings()
public void flush()
throws BulkInserter.InsertException
BulkInserter.InsertException to
get the list of records that were being inserted if needed (for example,
to retry). Other queues may also still contain unflushed records if
this occurs.BulkInserter.InsertException - if an error occurs while insertingpublic void insert(T record) throws BulkInserter.InsertException
batch size, all records in the queue will be
inserted into GPUdb before the method returns. If an error occurs while
inserting the records, the records will no longer be in the queue nor in
GPUdb; catch BulkInserter.InsertException to get the list of records that were
being inserted if needed (for example, to retry).
Note: This version of insert(Object) will result in sequential
batch inserts in a background thread. Use insert(List) to allow
multiple queues to reach their batch size and parallelize all of those
batch inserts at once.record - the record to insertGPUdbException - if an error occurs while calculating shard/primary keysBulkInserter.InsertException - if an error occurs while insertingpublic void insert(List<T> records) throws BulkInserter.InsertException
batch size, all queues that have reached the
batch size will have their records inserted into
the database before the method returns. If an error occurs while
inserting the records, they will no longer be in that queue or the
database; catch BulkInserter.InsertException to get the list of records
that were being inserted (including any from the queue in question and
any remaining in the list not yet queued) if needed (for example, to
retry). Note that depending on the number of records, multiple calls to
the database may occur.
Note: This version of insert(List) will result in parallelizing
the batch inserts in background threads.records - the records to insertBulkInserter.InsertException - if an error occurs while insertingCopyright © 2025. All rights reserved.