com.gpudb.filesystem.common

Class FullFileBatchManager

java.lang.Object

com.gpudb.filesystem.common.FullFileBatchManager

public class FullFileBatchManager
extends Object

This is an internal class and not meant to be used by the end users of the filesystem API. The consequences of using this class directly in client code is not guaranteed and maybe undesirable. This class manages the task of creating batches of files where the files are candidates for one-shot uploads.

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,org.apache.commons.lang3.tuple.Pair<String,Long>>	fullFileNameToSizeMap This variable is used to maintain a map of file names in the local filesystem to the pair of values of target file name on the KIFS and the file size on the local filesystem.
protected List<Map<String,String>>	listOfFullFileNameToRemoteFileNameMap This variable is used to store the maps in a list so that each map has files whose sizes add up to a maximum of the value returned by the method getFileSizeToSplit() of the class GPUdbFileHandler.Options.

Method Summary

Modifier and Type	Method and Description
void	addFile(String localFileName, String remoteFileName, Long size) Add the details for a file to be uploaded one shot
void	clearBatches() Resets the internal data structures maintained for the batches
List<String>	getFullFileList() Gets the entire list of local file names across all batches
Map<String,String>	getNthBatch(int n) Gets a particular batch of files which could be uploaded one shot using a single call to the endpoint 'upload/files'.
int	getNumberOfBatches() Gets the number of batches of files which could be uploaded one shot

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fullFileNameToSizeMap

protected Map<String,org.apache.commons.lang3.tuple.Pair<String,Long>> fullFileNameToSizeMap

This variable is used to maintain a map of file names in the local filesystem to the pair of values of target file name on the KIFS and the file size on the local filesystem. This is used by the method FileOperation.sortFilesIntoFullAndMultipartLists(List, List). This method, after detecting a file to be a candidate for one shot upload will insert an entry into this map. This map is further split up into a list of maps and stored in the variable listOfFullFileNameToRemoteFileNameMap by the method createBatchesByMaxSum(). The purpose of splitting the map into the list of maps is to arrive at batches of full file uploads so that each batch has a cumulative size whose limit is given by the method getFileSizeToSplit() of the class GPUdbFileHandler.Options.

listOfFullFileNameToRemoteFileNameMap

protected List<Map<String,String>> listOfFullFileNameToRemoteFileNameMap

This variable is used to store the maps in a list so that each map has files whose sizes add up to a maximum of the value returned by the method getFileSizeToSplit() of the class GPUdbFileHandler.Options. Each entry in this list is a map of local file name to its corresponding KIFS file name. This is used by the method 'FileUploader#uploadFullFiles()'

Method Detail

clearBatches

public void clearBatches()

Resets the internal data structures maintained for the batches

getFullFileList

public List<String> getFullFileList()

Gets the entire list of local file names across all batches

Returns:

- list of local file names

addFile

public void addFile(String localFileName,
                    String remoteFileName,
                    Long size)

Add the details for a file to be uploaded one shot

Parameters:

localFileName - - Name of the file on the local filesystem

remoteFileName - - Name of the file on KIFS

size - - The size of the file, needed to partition the batches on

getNumberOfBatches

public int getNumberOfBatches()

Gets the number of batches of files which could be uploaded one shot

Returns:

- the number of batches

getNthBatch

public Map<String,String> getNthBatch(int n)

Gets a particular batch of files which could be uploaded one shot using a single call to the endpoint 'upload/files'.

Parameters:

n - - the number of the batch to retrieve

Returns:

- the Nth batch; this batch is a set of entries in a Map where each entry has the "local file name" as the key and the "name of the file on KIFS" as the value.