High-performance bulk inserter for Kinetica with support for multi-head ingest, More...

Inherits IAsyncDisposable, and IDisposable.

Public Member Functions
	BulkInserter (Kinetica kinetica, string tableName, KineticaType ktype, BulkInserterOptions? options=null)
	Creates a new BulkInserter for the specified table. More...

BackpressureMetrics	GetBackpressureMetrics ()
	Gets backpressure metrics. More...

void	Insert (T record)
	Inserts a single record. More...

async ValueTask	InsertAsync (T record, CancellationToken cancellationToken=default)
	Inserts a single record with async backpressure control. More...

void	InsertBatch (IReadOnlyList< T > records)
	Inserts multiple records with parallel routing computation. More...

async ValueTask	InsertBatchAsync (IReadOnlyList< T > records, CancellationToken cancellationToken=default)
	Inserts multiple records with async backpressure control. More...

void	Flush ()
	Flushes all queued records to Kinetica. More...

async Task	FlushAsync (CancellationToken cancellationToken=default)
	Flushes all queued records to Kinetica asynchronously. More...

List< InsertError >	DrainErrors ()
	Drains and returns all errors from the error queue. More...

IReadOnlyList< InsertError >	PeekErrors ()
	Peeks at errors without removing them. More...

void	Close ()
	Flushes all records and waits for all pending batches to complete. More...

async Task	CloseAsync (CancellationToken cancellationToken=default)
	Flushes all records and waits for all pending batches to complete. More...

void	Dispose ()
	Disposes the BulkInserter, closing it if not already closed. More...

async ValueTask	DisposeAsync ()
	Disposes the BulkInserter asynchronously. More...

Public Attributes
long	CountInserted => Interlocked.Read(ref _countInserted)
	Gets the total count of records inserted. More...

long	CountUpdated => Interlocked.Read(ref _countUpdated)
	Gets the total count of records updated. More...

long	PendingBatches => Interlocked.Read(ref _pendingBatches)
	Gets the number of batches currently pending (queued or in-flight). More...

long	TotalBatchesSent => Interlocked.Read(ref _totalBatchesSent)
	Gets the total number of batches sent. More...

long	TotalBatchesFailed => Interlocked.Read(ref _totalBatchesFailed)
	Gets the total number of batches that failed. More...

string	TableName => _tableName
	Gets the table name. More...

int	NumWorkers => _numWorkers
	Gets the number of workers. More...

int	ErrorCount => _errorCount
	Gets the number of errors in the error queue. More...

bool	IsTimedFlushRunning => _timedFlushRunning
	Gets whether timed flush is currently running. More...

int	NumClusterSwitches => _numClusterSwitches
	Gets the number of times the cluster has been switched due to failover. More...

int	HARingSize => _dbHaRingSize
	Gets the HA ring size (number of clusters). More...

Detailed Description

High-performance bulk inserter for Kinetica with support for multi-head ingest,

parallel processing, backpressure control, and async flush workers.

This implementation follows the Rust BulkInserter architecture with:

Stripe-based worker queues to reduce lock contention
Parallel batch processing using TPL
Async flush workers with bounded channels for backpressure
Batch insertion callbacks for monitoring
Automatic retry with exponential backoff

Types must implement IShardKeyExtractor for shard-aware routing.

Template Parameters

T	The record type to insert. Must implement IShardKeyExtractor.

Type Constraints

T

:

Records.IShardKeyExtractor

Definition at line 28 of file BulkInserter.cs.

Constructor & Destructor Documentation

◆ BulkInserter()

BulkInserter< T >.BulkInserter	(	Kinetica	kinetica,
		string	tableName,
		KineticaType	ktype,
		BulkInserterOptions?	options = `null`
	)

inline

Creates a new BulkInserter for the specified table.

Parameters

kinetica	The Kinetica connection.
tableName	The target table name.
ktype	The KineticaType for the table.
options	Optional configuration options.

Definition at line 104 of file BulkInserter.cs.

Member Function Documentation

◆ Close()

void BulkInserter< T >.Close ( )

inline

Flushes all records and waits for all pending batches to complete.

Definition at line 1129 of file BulkInserter.cs.

◆ CloseAsync()

async Task BulkInserter< T >.CloseAsync ( CancellationToken cancellationToken = default )

inline

Flushes all records and waits for all pending batches to complete.

This method will wait indefinitely for all batches to be sent to prevent data loss.

Parameters

cancellationToken Cancellation token.

Definition at line 1139 of file BulkInserter.cs.

◆ Dispose()

void BulkInserter< T >.Dispose ( )

inline

Disposes the BulkInserter, closing it if not already closed.

Definition at line 1205 of file BulkInserter.cs.

◆ DisposeAsync()

async ValueTask BulkInserter< T >.DisposeAsync ( )

inline

Disposes the BulkInserter asynchronously.

Definition at line 1213 of file BulkInserter.cs.

◆ DrainErrors()

List<InsertError> BulkInserter< T >.DrainErrors ( )

inline

Drains and returns all errors from the error queue.

Definition at line 875 of file BulkInserter.cs.

◆ Flush()

void BulkInserter< T >.Flush ( )

inline

Flushes all queued records to Kinetica.

Definition at line 580 of file BulkInserter.cs.

◆ FlushAsync()

async Task BulkInserter< T >.FlushAsync ( CancellationToken cancellationToken = default )

inline

Flushes all queued records to Kinetica asynchronously.

Parameters

cancellationToken Cancellation token.

Definition at line 589 of file BulkInserter.cs.

◆ GetBackpressureMetrics()

BackpressureMetrics BulkInserter< T >.GetBackpressureMetrics ( )

inline

Gets backpressure metrics.

Definition at line 317 of file BulkInserter.cs.

◆ Insert()

void BulkInserter< T >.Insert ( T record )

inline

Inserts a single record.

Non-blocking unless backpressure is applied.

Parameters

record The record to insert.

Definition at line 340 of file BulkInserter.cs.

◆ InsertAsync()

async ValueTask BulkInserter< T >.InsertAsync	(	T	record,
		CancellationToken	cancellationToken = `default`
	)

inline

Inserts a single record with async backpressure control.

Waits if the maximum number of in-flight batches is reached.

Parameters

record	The record to insert.
cancellationToken	Cancellation token.

Definition at line 361 of file BulkInserter.cs.

◆ InsertBatch()

void BulkInserter< T >.InsertBatch ( IReadOnlyList< T > records )

inline

Inserts multiple records with parallel routing computation.

This is the most efficient method for bulk inserts.

Parameters

records The records to insert.

Definition at line 383 of file BulkInserter.cs.

◆ InsertBatchAsync()

async ValueTask BulkInserter< T >.InsertBatchAsync	(	IReadOnlyList< T >	records,
		CancellationToken	cancellationToken = `default`
	)

inline

Inserts multiple records with async backpressure control.

Parameters

records	The records to insert.
cancellationToken	Cancellation token.

Definition at line 450 of file BulkInserter.cs.

◆ PeekErrors()

IReadOnlyList<InsertError> BulkInserter< T >.PeekErrors ( )

inline

Peeks at errors without removing them.

Definition at line 889 of file BulkInserter.cs.

Member Data Documentation

◆ CountInserted

long BulkInserter< T >.CountInserted => Interlocked.Read(ref _countInserted)

Gets the total count of records inserted.

Definition at line 272 of file BulkInserter.cs.

◆ CountUpdated

long BulkInserter< T >.CountUpdated => Interlocked.Read(ref _countUpdated)

Gets the total count of records updated.

Definition at line 277 of file BulkInserter.cs.

◆ ErrorCount

int BulkInserter< T >.ErrorCount => _errorCount

Gets the number of errors in the error queue.

Definition at line 307 of file BulkInserter.cs.

◆ HARingSize

int BulkInserter< T >.HARingSize => _dbHaRingSize

Gets the HA ring size (number of clusters).

Definition at line 906 of file BulkInserter.cs.

◆ IsTimedFlushRunning

bool BulkInserter< T >.IsTimedFlushRunning => _timedFlushRunning

Gets whether timed flush is currently running.

Definition at line 312 of file BulkInserter.cs.

◆ NumClusterSwitches

int BulkInserter< T >.NumClusterSwitches => _numClusterSwitches

Gets the number of times the cluster has been switched due to failover.

Definition at line 901 of file BulkInserter.cs.

◆ NumWorkers

int BulkInserter< T >.NumWorkers => _numWorkers

Gets the number of workers.

Definition at line 302 of file BulkInserter.cs.

◆ PendingBatches

long BulkInserter< T >.PendingBatches => Interlocked.Read(ref _pendingBatches)

Gets the number of batches currently pending (queued or in-flight).

Definition at line 282 of file BulkInserter.cs.

◆ TableName

string BulkInserter< T >.TableName => _tableName

Gets the table name.

Definition at line 297 of file BulkInserter.cs.

◆ TotalBatchesFailed

long BulkInserter< T >.TotalBatchesFailed => Interlocked.Read(ref _totalBatchesFailed)

Gets the total number of batches that failed.

Definition at line 292 of file BulkInserter.cs.

◆ TotalBatchesSent

long BulkInserter< T >.TotalBatchesSent => Interlocked.Read(ref _totalBatchesSent)

Gets the total number of batches sent.

Definition at line 287 of file BulkInserter.cs.

The documentation for this class was generated from the following file:

Kinetica/BulkInserter/BulkInserter.cs

Public Member Functions

Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ BulkInserter()

Member Function Documentation

◆ Close()

◆ CloseAsync()

◆ Dispose()

◆ DisposeAsync()

◆ DrainErrors()

◆ Flush()

◆ FlushAsync()

◆ GetBackpressureMetrics()

◆ Insert()

◆ InsertAsync()

◆ InsertBatch()

◆ InsertBatchAsync()

◆ PeekErrors()

Member Data Documentation

◆ CountInserted

◆ CountUpdated

◆ ErrorCount

◆ HARingSize

◆ IsTimedFlushRunning

◆ NumClusterSwitches

◆ NumWorkers

◆ PendingBatches

◆ TableName

◆ TotalBatchesFailed

◆ TotalBatchesSent