High-performance bulk inserter for Kinetica with support for multi-head ingest, parallel processing, backpressure control, and async flush workers. More…
Inherits IAsyncDisposable, IDisposable, IAsyncDisposable, and IDisposable.
Public Member Functions | |
| BulkInserter (Kinetica kinetica, string tableName, KineticaType ktype, BulkInserterOptions? options=null) | |
| Creates a new BulkInserter for the specified table. | |
| BackpressureMetrics | GetBackpressureMetrics () |
| Gets backpressure metrics. | |
| void | Insert (T record) |
| Inserts a single record. | |
| async ValueTask | InsertAsync (T record, CancellationToken cancellationToken=default) |
| Inserts a single record with async backpressure control. | |
| void | InsertBatch (IReadOnlyList< T > records) |
| Inserts multiple records with parallel routing computation. | |
| async ValueTask | InsertBatchAsync (IReadOnlyList< T > records, CancellationToken cancellationToken=default) |
| Inserts multiple records with async backpressure control. | |
| void | Flush () |
| Flushes all queued records to Kinetica. | |
| async Task | FlushAsync (CancellationToken cancellationToken=default) |
| Flushes all queued records to Kinetica asynchronously. | |
| List< InsertError > | DrainErrors () |
| Drains and returns all errors from the error queue. | |
| IReadOnlyList< InsertError > | PeekErrors () |
| Peeks at errors without removing them. | |
| void | Close () |
| Flushes all records and waits for all pending batches to complete. | |
| async Task | CloseAsync (CancellationToken cancellationToken=default) |
| Flushes all records and waits for all pending batches to complete. | |
| void | Dispose () |
| Disposes the BulkInserter, closing it if not already closed. | |
| async ValueTask | DisposeAsync () |
| Disposes the BulkInserter asynchronously. | |
| FlushJob (Uri workerUrl, int workerIndex, int stripeIndex, IReadOnlyList< T > records, bool usedBackpressure=false) | |
| BulkInserter (Kinetica kinetica, string tableName, KineticaType ktype, BulkInserterOptions? options=null) | |
| Creates a new BulkInserter for the specified table. | |
| BackpressureMetrics | GetBackpressureMetrics () |
| Gets backpressure metrics. | |
| void | Insert (T record) |
| Inserts a single record. | |
| async ValueTask | InsertAsync (T record, CancellationToken cancellationToken=default) |
| Inserts a single record with async backpressure control. | |
| void | InsertBatch (IReadOnlyList< T > records) |
| Inserts multiple records with parallel routing computation. | |
| async ValueTask | InsertBatchAsync (IReadOnlyList< T > records, CancellationToken cancellationToken=default) |
| Inserts multiple records with async backpressure control. | |
| void | Flush () |
| Flushes all queued records to Kinetica. | |
| async Task | FlushAsync (CancellationToken cancellationToken=default) |
| Flushes all queued records to Kinetica asynchronously. | |
| List< InsertError > | DrainErrors () |
| Drains and returns all errors from the error queue. | |
| IReadOnlyList< InsertError > | PeekErrors () |
| Peeks at errors without removing them. | |
| void | Close () |
| Flushes all records and waits for all pending batches to complete. | |
| async Task | CloseAsync (CancellationToken cancellationToken=default) |
| Flushes all records and waits for all pending batches to complete. | |
| void | Dispose () |
| Disposes the BulkInserter, closing it if not already closed. | |
| async ValueTask | DisposeAsync () |
| Disposes the BulkInserter asynchronously. | |
| FlushJob (Uri workerUrl, int workerIndex, int stripeIndex, IReadOnlyList< T > records, bool usedBackpressure=false) | |
Properties | |
| long | CountInserted [get] |
| Gets the total count of records inserted. | |
| long | CountUpdated [get] |
| Gets the total count of records updated. | |
| long | PendingBatches [get] |
| Gets the number of batches currently pending (queued or in-flight). | |
| long | TotalBatchesSent [get] |
| Gets the total number of batches sent. | |
| long | TotalBatchesFailed [get] |
| Gets the total number of batches that failed. | |
| string | TableName [get] |
| Gets the table name. | |
| int | NumWorkers [get] |
| Gets the number of workers. | |
| int | ErrorCount [get] |
| Gets the number of errors in the error queue. | |
| long | TotalRecordsQueued [get] |
| Gets the total number of records queued for insertion. | |
| long | TotalRecordsSubmitted [get] |
| Gets the total number of records submitted to the server (in batches). | |
| bool | IsTimedFlushRunning [get] |
| Gets whether timed flush is currently running. | |
| int | NumClusterSwitches [get] |
| Gets the number of times the cluster has been switched due to failover. | |
| int | HARingSize [get] |
| Gets the HA ring size (number of clusters). | |
| Uri | WorkerUrl [get] |
| int | WorkerIndex [get] |
| int | StripeIndex [get] |
| IReadOnlyList< T > | Records [get] |
| bool | UsedBackpressure [get] |
Detailed Description
High-performance bulk inserter for Kinetica with support for multi-head ingest, parallel processing, backpressure control, and async flush workers.
This implementation follows the Rust BulkInserter architecture with:
- Stripe-based worker queues to reduce lock contention
- Parallel batch processing using TPL
- Async flush workers with bounded channels for backpressure
- Batch insertion callbacks for monitoring
- Automatic retry with exponential backoff
Types must implement IShardKeyExtractor for shard-aware routing.
| T | The record type to insert. Must implement IShardKeyExtractor. |
| T | : | Records.IShardKeyExtractor |
Definition at line 23 of file BulkInserter.cs.
Member Function Documentation
◆ BulkInserter() [1/2]
| inline |
Creates a new BulkInserter for the specified table.
| kinetica | The Kinetica connection. |
| tableName | The target table name. |
| ktype | The KineticaType for the table. |
| options | Optional configuration options. |
Definition at line 102 of file BulkInserter.cs.
◆ BulkInserter() [2/2]
| inline |
Creates a new BulkInserter for the specified table.
| kinetica | The Kinetica connection. |
| tableName | The target table name. |
| ktype | The KineticaType for the table. |
| options | Optional configuration options. |
Definition at line 102 of file BulkInserter.cs.
◆ Close() [1/2]
| inline |
Flushes all records and waits for all pending batches to complete.
Definition at line 1194 of file BulkInserter.cs.
◆ Close() [2/2]
| inline |
Flushes all records and waits for all pending batches to complete.
Definition at line 1194 of file BulkInserter.cs.
◆ CloseAsync() [1/2]
| inline |
Flushes all records and waits for all pending batches to complete.
This method will wait indefinitely for all batches to be sent to prevent data loss.
| cancellationToken | Cancellation token. |
Definition at line 1204 of file BulkInserter.cs.
◆ CloseAsync() [2/2]
| inline |
Flushes all records and waits for all pending batches to complete.
This method will wait indefinitely for all batches to be sent to prevent data loss.
| cancellationToken | Cancellation token. |
Definition at line 1204 of file BulkInserter.cs.
◆ Dispose() [1/2]
| inline |
Disposes the BulkInserter, closing it if not already closed.
Definition at line 1276 of file BulkInserter.cs.
◆ Dispose() [2/2]
| inline |
Disposes the BulkInserter, closing it if not already closed.
Definition at line 1276 of file BulkInserter.cs.
◆ DisposeAsync() [1/2]
| inline |
Disposes the BulkInserter asynchronously.
Definition at line 1284 of file BulkInserter.cs.
◆ DisposeAsync() [2/2]
| inline |
Disposes the BulkInserter asynchronously.
Definition at line 1284 of file BulkInserter.cs.
◆ DrainErrors() [1/2]
| inline |
Drains and returns all errors from the error queue.
Definition at line 940 of file BulkInserter.cs.
◆ DrainErrors() [2/2]
| inline |
Drains and returns all errors from the error queue.
Definition at line 940 of file BulkInserter.cs.
◆ Flush() [1/2]
| inline |
Flushes all queued records to Kinetica.
Definition at line 615 of file BulkInserter.cs.
◆ Flush() [2/2]
| inline |
Flushes all queued records to Kinetica.
Definition at line 615 of file BulkInserter.cs.
◆ FlushAsync() [1/2]
| inline |
Flushes all queued records to Kinetica asynchronously.
| cancellationToken | Cancellation token. |
Definition at line 624 of file BulkInserter.cs.
◆ FlushAsync() [2/2]
| inline |
Flushes all queued records to Kinetica asynchronously.
| cancellationToken | Cancellation token. |
Definition at line 624 of file BulkInserter.cs.
◆ FlushJob() [1/2]
| inline |
Definition at line 1324 of file BulkInserter.cs.
◆ FlushJob() [2/2]
| inline |
Definition at line 1324 of file BulkInserter.cs.
◆ GetBackpressureMetrics() [1/2]
| inline |
Gets backpressure metrics.
Definition at line 337 of file BulkInserter.cs.
◆ GetBackpressureMetrics() [2/2]
| inline |
Gets backpressure metrics.
Definition at line 337 of file BulkInserter.cs.
◆ Insert() [1/2]
| inline |
Inserts a single record.
Non-blocking unless backpressure is applied.
| record | The record to insert. |
| ObjectDisposedException | If the inserter has been disposed. |
| InvalidOperationException | If the inserter has been closed. |
Definition at line 362 of file BulkInserter.cs.
◆ Insert() [2/2]
| inline |
Inserts a single record.
Non-blocking unless backpressure is applied.
| record | The record to insert. |
| ObjectDisposedException | If the inserter has been disposed. |
| InvalidOperationException | If the inserter has been closed. |
Definition at line 362 of file BulkInserter.cs.
◆ InsertAsync() [1/2]
| inline |
Inserts a single record with async backpressure control.
Waits if the maximum number of in-flight batches is reached.
| record | The record to insert. |
| cancellationToken | Cancellation token. |
| ObjectDisposedException | If the inserter has been disposed. |
| InvalidOperationException | If the inserter has been closed. |
Definition at line 388 of file BulkInserter.cs.
◆ InsertAsync() [2/2]
| inline |
Inserts a single record with async backpressure control.
Waits if the maximum number of in-flight batches is reached.
| record | The record to insert. |
| cancellationToken | Cancellation token. |
| ObjectDisposedException | If the inserter has been disposed. |
| InvalidOperationException | If the inserter has been closed. |
Definition at line 388 of file BulkInserter.cs.
◆ InsertBatch() [1/2]
| inline |
Inserts multiple records with parallel routing computation.
This is the most efficient method for bulk inserts.
| records | The records to insert. |
| ObjectDisposedException | If the inserter has been disposed. |
| InvalidOperationException | If the inserter has been closed. |
Definition at line 415 of file BulkInserter.cs.
◆ InsertBatch() [2/2]
| inline |
Inserts multiple records with parallel routing computation.
This is the most efficient method for bulk inserts.
| records | The records to insert. |
| ObjectDisposedException | If the inserter has been disposed. |
| InvalidOperationException | If the inserter has been closed. |
Definition at line 415 of file BulkInserter.cs.
◆ InsertBatchAsync() [1/2]
| inline |
Inserts multiple records with async backpressure control.
| records | The records to insert. |
| cancellationToken | Cancellation token. |
Definition at line 485 of file BulkInserter.cs.
◆ InsertBatchAsync() [2/2]
| inline |
Inserts multiple records with async backpressure control.
| records | The records to insert. |
| cancellationToken | Cancellation token. |
Definition at line 485 of file BulkInserter.cs.
◆ PeekErrors() [1/2]
| inline |
Peeks at errors without removing them.
Definition at line 954 of file BulkInserter.cs.
◆ PeekErrors() [2/2]
| inline |
Peeks at errors without removing them.
Definition at line 954 of file BulkInserter.cs.
Property Documentation
◆ CountInserted
| get |
Gets the total count of records inserted.
Definition at line 282 of file BulkInserter.cs.
◆ CountUpdated
| get |
Gets the total count of records updated.
Definition at line 287 of file BulkInserter.cs.
◆ ErrorCount
| get |
Gets the number of errors in the error queue.
Definition at line 317 of file BulkInserter.cs.
◆ HARingSize
| get |
Gets the HA ring size (number of clusters).
Definition at line 971 of file BulkInserter.cs.
◆ IsTimedFlushRunning
| get |
Gets whether timed flush is currently running.
Definition at line 332 of file BulkInserter.cs.
◆ NumClusterSwitches
| get |
Gets the number of times the cluster has been switched due to failover.
Definition at line 966 of file BulkInserter.cs.
◆ NumWorkers
| get |
Gets the number of workers.
Definition at line 312 of file BulkInserter.cs.
◆ PendingBatches
| get |
Gets the number of batches currently pending (queued or in-flight).
Definition at line 292 of file BulkInserter.cs.
◆ Records
| get |
Definition at line 1321 of file BulkInserter.cs.
◆ StripeIndex
| get |
Definition at line 1320 of file BulkInserter.cs.
◆ TableName
| get |
Gets the table name.
Definition at line 307 of file BulkInserter.cs.
◆ TotalBatchesFailed
| get |
Gets the total number of batches that failed.
Definition at line 302 of file BulkInserter.cs.
◆ TotalBatchesSent
| get |
Gets the total number of batches sent.
Definition at line 297 of file BulkInserter.cs.
◆ TotalRecordsQueued
| get |
Gets the total number of records queued for insertion.
Definition at line 322 of file BulkInserter.cs.
◆ TotalRecordsSubmitted
| get |
Gets the total number of records submitted to the server (in batches).
Definition at line 327 of file BulkInserter.cs.
◆ UsedBackpressure
| get |
Definition at line 1322 of file BulkInserter.cs.
◆ WorkerIndex
| get |
Definition at line 1319 of file BulkInserter.cs.
◆ WorkerUrl
| get |
Definition at line 1318 of file BulkInserter.cs.
The documentation for this class was generated from the following files:
- _build/public-os_ubuntu24.04-arch_amd64-cc_gcc_13.3.0/install/Kinetica/BulkInserter/BulkInserter.cs
- Kinetica/BulkInserter/BulkInserter.cs