com.gpudb.filesystem.common

Class FileOperation

java.lang.Object

com.gpudb.filesystem.common.FileOperation

Direct Known Subclasses:

FileDownloader, FileUploader

public class FileOperation
extends Object

This is an internal class and not meant to be used by the end users of the filesystem API. The consequences of using this class directly in client code is not guaranteed and maybe undesirable. This is the base class from which the classes FileUploader and FileDownloader are derived. The purpose of this class is to model certain basic functions like searching directories for specific file patterns, retrieving file attributes like size, creation time etc. for both files on the KIFS and local directories and also determining which files to be uploaded/downloaded either in one go or in parts based on the size and a threshold which is given by the method getFileSizeToSplit() of the GPUdbFileHandler.Options class.

Field Summary

Fields

Modifier and Type	Field and Description
protected GPUdb	db
protected String	dirName This stores the directory name into which the source files for any operation (upload or download) will be written.
protected GPUdbFileHandler.Options	fileHandlerOptions
protected List<String>	fileNames The list of file names to be either uploaded or downloaded.
protected FullFileBatchManager	fullFileBatchManager
protected List<String>	fullFileList List of file names that can be downloaded in full
protected List<String>	fullRemoteFileNames
protected List<String>	multiPartList List of file names that are multi part uploads/downloads
protected List<String>	multiPartRemoteFileNames
protected Set<String>	namesOfFilesUploaded The list of files uploaded, used by the FileIngestor to insert records from the uploaded files into the database.
protected OpMode	opMode
protected boolean	recursive Indicates whether the search for files is to be recursive through a directory hierarchy or not.

Constructor Summary

Constructors

Constructor and Description
FileOperation(GPUdb db, OpMode opMode, List<String> fileNames, String dirName, boolean recursive, GPUdbFileHandler.Options fileHandlerOptions) Constructs a new file operation instance, managing the transfer of a set of files to a target directory.

Method Summary

Modifier and Type	Method and Description
protected void	decideMultiPart() This method resolves the file names passed in and decides on which could be downloaded in one go and which ones are large enough to be downloaded in parts.
protected List<KifsFileInfo>	getFileInfoFromServer(String path) This method gets the file stats for the files residing on the KIFS.
static String	getKifsPathSeparator() Deprecated.
Set<String>	getNamesOfFilesUploaded() This method returns the list of file names uploaded without the path information.
static boolean	localDirExists(String localDirName) This is a utility method that checks if a local directory exists or not.
static boolean	localFileExists(String localFileName) This method checks if a local file exists or not.
static List<org.apache.commons.lang3.tuple.Triple<String,String,String>>	parseFileNames(List<String> fileNamesToParse) This method parses the list of file names passed in to the constructor of the classes FileUploader, FileDownloader which are derivatives of this class.
protected void	sortFilesIntoFullAndMultipartLists(List<String> fileList, List<String> remoteFileList) This method takes a list of completely resolved file paths and puts each file into either the multiPartList or fullFileList depending on the file size threshold returned by GPUdbFileHandler.Options.getFileSizeToSplit().
protected org.apache.commons.lang3.tuple.Pair<List<String>,List<String>>	traverseLocalDirectories(String startingDirName, List<String> localFileNames) This method traverses the local directory hierarchy starting with the directory given by the argument startingDirName and the list of file names.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

opMode

protected final OpMode opMode

db

protected final GPUdb db

recursive

protected final boolean recursive

Indicates whether the search for files is to be recursive through a directory hierarchy or not. This applies to only uploads for the time being and the search through local directories could be recursive.

fileNames

protected List<String> fileNames

The list of file names to be either uploaded or downloaded. For download, each element in the list must have the full KIFS path and for upload each element must have the full local path.

namesOfFilesUploaded

protected Set<String> namesOfFilesUploaded

The list of files uploaded, used by the FileIngestor to insert records from the uploaded files into the database.

dirName

protected String dirName

This stores the directory name into which the source files for any operation (upload or download) will be written. So, for uploads, this is the name of the KIFS directory; and for downloads, this is the name of the local directory.

multiPartList

protected List<String> multiPartList

List of file names that are multi part uploads/downloads

multiPartRemoteFileNames

protected List<String> multiPartRemoteFileNames

fullFileList

protected List<String> fullFileList

List of file names that can be downloaded in full

fullRemoteFileNames

protected List<String> fullRemoteFileNames

fullFileBatchManager

protected FullFileBatchManager fullFileBatchManager

fileHandlerOptions

protected final GPUdbFileHandler.Options fileHandlerOptions

Constructor Detail

FileOperation

public FileOperation(GPUdb db,
                     OpMode opMode,
                     List<String> fileNames,
                     String dirName,
                     boolean recursive,
                     GPUdbFileHandler.Options fileHandlerOptions)
              throws GPUdbException

Constructs a new file operation instance, managing the transfer of a set of files to a target directory.

Parameters:

db - The GPUdb instance used to access KiFS.

opMode - Indicates whether this is an upload or download operation.

fileNames - List of source file names.

dirName - Name of the local/remote target directory depending upon whether it is a download/upload operation.

recursive - Indicates whether any directories given in fileNames should be searched for files recursively.

fileHandlerOptions - Options for setting up the files for transfer.

Throws:

GPUdbException - propagates exceptions raised from various argument validations.

Method Detail

getNamesOfFilesUploaded

public Set<String> getNamesOfFilesUploaded()

This method returns the list of file names uploaded without the path information. This used by the FileIngestor.ingestFromFiles() method to prepare the list of file names to be passed on to the GPUdb.insertRecordsFromFiles(InsertRecordsFromFilesRequest) endpoint.

Returns:

A set of file names without the path information.

decideMultiPart

protected void decideMultiPart()
                        throws GPUdbException

This method resolves the file names passed in and decides on which could be downloaded in one go and which ones are large enough to be downloaded in parts. This method calls traverseLocalDirectories(String, List) and gets the file names properly resolved in a list.

Throws:

GPUdbException

sortFilesIntoFullAndMultipartLists

protected void sortFilesIntoFullAndMultipartLists(List<String> fileList,
                                                  List<String> remoteFileList)

This method takes a list of completely resolved file paths and puts each file into either the multiPartList or fullFileList depending on the file size threshold returned by GPUdbFileHandler.Options.getFileSizeToSplit().

Parameters:

fileList - A list of source file names to triage by size.

remoteFileList - A list of target file names for the given sources.

getFileInfoFromServer

protected List<KifsFileInfo> getFileInfoFromServer(String path)
                                            throws GPUdbException

This method gets the file stats for the files residing on the KIFS.

Parameters:

path - Name of the KIFS file or directory of files to retrieve info on.

Returns:

List of KifsFileInfo objects for the KiFS file(s) found.

Throws:

GPUdbException - If the '/show/files' endpoint call fails.

traverseLocalDirectories

protected org.apache.commons.lang3.tuple.Pair<List<String>,List<String>> traverseLocalDirectories(String startingDirName,
                                                                                                  List<String> localFileNames)
                                                                                           throws IOException

This method traverses the local directory hierarchy starting with the directory given by the argument startingDirName and the list of file names. The method takes care of recursive searches based on a pattern like (*|**|?) and returns the list of all files found with their fully normalized and absolute paths for the other methods to work with. This method is internal and is only invoked when the input file names have a wildcard pattern.

Parameters:

startingDirName - Local directory to start searching from

localFileNames - Local file names & wildcard patterns to search for

Returns:

Pair of lists; the first element is the list of local file names and the second a list of remote file names

Throws:

IOException

localDirExists

public static boolean localDirExists(String localDirName)

This is a utility method that checks if a local directory exists or not.

Parameters:

localDirName - Name of the local directory to check for.

Returns:

True if the directory exists, and false if it doesn't exist or if the localDirName is null or empty.

localFileExists

public static boolean localFileExists(String localFileName)

This method checks if a local file exists or not. If the file name is a wildcard pattern it skips the check.

Parameters:

localFileName - Name of the file.

Returns:

True if the file exists, false otherwise.

getKifsPathSeparator

public static String getKifsPathSeparator()

Deprecated.

Use GPUdbFileHandler.KIFS_PATH_SEPARATOR directly instead.

Returns:

The separator character used by KiFS between a directory and the files it contains; can also be used in file names to create "virtual" subdirectories.

parseFileNames

public static List<org.apache.commons.lang3.tuple.Triple<String,String,String>> parseFileNames(List<String> fileNamesToParse)

This method parses the list of file names passed in to the constructor of the classes FileUploader, FileDownloader which are derivatives of this class. It resolves the file names, normalizes them and returns a corresponding list of absolute paths which are used by the methods of this class.

Parameters:

fileNamesToParse - List of file names passed in to the constructor of the derivatives of this class.

Returns:

A list of Triple objects where the first element is the root of the file path, the second the full path without the file name and the third just the file name itself.