com.gpudb.filesystem.common

Class FileOperation

java.lang.Object

com.gpudb.filesystem.common.FileOperation

Direct Known Subclasses:

FileDownloader, FileUploader

public class FileOperation
extends Object

This is an internal class and not meant to be used by the end users of the filesystem API. The consequences of using this class directly in client code is not guaranteed and maybe undesirable. This is the base class from which the classes FileUploader and FileDownloader are derived. The purpose of this class is to model certain basic functions like searching directories for specific file patterns, retrieving file attributes like size, creation time etc. for both files on the KIFS and local directories and also determining which files to be uploaded/downloaded either in one go or in parts based on the size and a threshold which is given by the method getFileSizeToSplit() of the GPUdbFileHandler.Options class.

Field Summary

Fields

Modifier and Type	Field and Description
protected GPUdb	db
protected String	dirName This stores the directory name from which the source files for any operation (upload or download) is to be searched in.
protected GPUdbFileHandler.Options	fileHandlerOptions
protected List<String>	fileNames The list of file names to be either uploaded or downloaded.
protected FullFileBatchManager	fullFileBatchManager
protected List<String>	fullFileList List of file names that can be downloaded in full
protected List<String>	fullRemoteFileNames
protected static String	KIFS_PATH_SEPARATOR
protected List<String>	multiPartList List of file names that are multi part uploads/downloads
protected List<String>	multiPartRemoteFileNames
protected Set<String>	namesOfFilesUploaded
protected OpMode	opMode
protected boolean	recursive Indicates whether the search for files is to be recursive through a directory hierarchy or not.

Constructor Summary

Constructors

Constructor and Description
FileOperation(GPUdb db, OpMode opMode, List<String> fileNames, String dirName, boolean recursive, GPUdbFileHandler.Options fileHandlerOptions)

Method Summary

Modifier and Type	Method and Description
protected void	decideMultiPart(OpMode opMode) This method resolves the file names passed in and decides on which could be downloaded in one go and which ones are large enough to be downloaded in parts.
protected List<KifsFileInfo>	getFileInfoFromServer(String dirName) This method gets the file stats for the files residing on the KIFS.
protected List<KifsFileInfo>	getFileInfoFromServer(String dirName, String fileName) This method gets the file stats for the files residing on the KIFS.
static String	getKifsPathSeparator()
Set<String>	getNamesOfFilesUploaded() This method returns the list of file names uploaded without the path information.
static boolean	localDirExists(String dirName) This is an utility method that checks if a local directory exists or not
static boolean	localFileExists(String fileName) This method checks if a local file exists or not.
static List<org.apache.commons.lang3.tuple.Triple<String,String,String>>	parseFileNames(List<String> fileNames) This method parses the list of file names passed in to the constructor of the classes FileUploader, FileDownloader which are derivatives of this class.
protected void	sortFilesIntoFullAndMultipartLists(List<String> fileList, List<String> remoteFileList) This method takes a list of completely resolved file paths and puts each file into either the multiPartList or fullFileList depending on the file size threshold returned by the method getFileSizeToSplit() of the GPUdbFileHandler.Options class.
protected org.apache.commons.lang3.tuple.Pair<List<String>,List<String>>	traverseLocalDirectories(String startingDirName, List<String> fileNames) This method traverses the local directory hierarchy starting with the directory given by the argument 'startingDirName' and the list of file names.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KIFS_PATH_SEPARATOR

protected static final String KIFS_PATH_SEPARATOR

See Also:

Constant Field Values

opMode

protected final OpMode opMode

db

protected final GPUdb db

recursive

protected final boolean recursive

Indicates whether the search for files is to be recursive through a directory hierarchy or not. This applies to only uploads for the time being and the search through local directories could be recursive.

fileNames

protected List<String> fileNames

The list of file names to be either uploaded or downloaded. For download, each element in the list must have the full KIFS path and for upload each element must have the full local path.

namesOfFilesUploaded

protected Set<String> namesOfFilesUploaded

dirName

protected String dirName

This stores the directory name from which the source files for any operation (upload or download) is to be searched in. So, for uploads this is the name of the KIFS directory and for downloads this is the name of the local directory.

multiPartList

protected List<String> multiPartList

List of file names that are multi part uploads/downloads

multiPartRemoteFileNames

protected List<String> multiPartRemoteFileNames

fullFileList

protected List<String> fullFileList

List of file names that can be downloaded in full

fullRemoteFileNames

protected List<String> fullRemoteFileNames

fullFileBatchManager

protected FullFileBatchManager fullFileBatchManager

fileHandlerOptions

protected final GPUdbFileHandler.Options fileHandlerOptions

Constructor Detail

FileOperation

public FileOperation(GPUdb db,
                     OpMode opMode,
                     List<String> fileNames,
                     String dirName,
                     boolean recursive,
                     GPUdbFileHandler.Options fileHandlerOptions)
              throws GPUdbException

Parameters:

db - - The GPUdb instance

opMode - - Indicates whether this is an upload or download operation

fileNames - - List - List of inout file names

dirName - - Name of the local/remote directory depending upon whether it is a download/upload operation.

recursive - - boolean - Indicates whether the path resolution needs

fileHandlerOptions - - a GPUdbFileHandler.Options type object

Throws:

GPUdbException - - propagates exceptions raised from various argument validations.

Method Detail

getNamesOfFilesUploaded

public Set<String> getNamesOfFilesUploaded()

This method returns the list of file names uploaded without the path information. This used by the FileIngestor.ingestFromFiles() method to prepare the list of file names to be passed on to the GPUdb.insertRecordsFromFiles(InsertRecordsFromFilesRequest) endpoint.

Returns:

- A set of file names without the path information.

decideMultiPart

protected void decideMultiPart(OpMode opMode)
                        throws GPUdbException

This method resolves the file names passed in and decides on which could be downloaded in one go and which ones are large enough to be downloaded in parts. This method internally calls the method traverseLocalDirectories(String, List) and gets the file names properly resolved in a list.

Parameters:

opMode - - OpMode

Throws:

GPUdbException

sortFilesIntoFullAndMultipartLists

protected void sortFilesIntoFullAndMultipartLists(List<String> fileList,
                                                  List<String> remoteFileList)

This method takes a list of completely resolved file paths and puts each file into either the multiPartList or fullFileList depending on the file size threshold returned by the method getFileSizeToSplit() of the GPUdbFileHandler.Options class.

Parameters:

fileList - - a list of file names

getFileInfoFromServer

protected List<KifsFileInfo> getFileInfoFromServer(String dirName,
                                                   String fileName)
                                            throws GPUdbException

This method gets the file stats for the files residing on the KIFS.

Parameters:

dirName - - Name of the KIFS directory root.

Returns:

- List of KifsFileInfo objects

Throws:

GPUdbException - if 'show/files' endpoint fails

getFileInfoFromServer

protected List<KifsFileInfo> getFileInfoFromServer(String dirName)
                                            throws GPUdbException

This method gets the file stats for the files residing on the KIFS.

Parameters:

dirName - - Name of the KIFS directory root.

Returns:

- List of KifsFileInfo objects

Throws:

GPUdbException - in case 'show/files' endpoint fails

traverseLocalDirectories

protected org.apache.commons.lang3.tuple.Pair<List<String>,List<String>> traverseLocalDirectories(String startingDirName,
                                                                                                  List<String> fileNames)
                                                                                           throws IOException

This method traverses the local directory hierarchy starting with the directory given by the argument 'startingDirName' and the list of file names. The method takes care of recursive searches based on a pattern like (*|**|?) and returns the list of all files found with their fully normalized and absolute paths for the other methods to work with. This method is internal and is only invoked when the input file names have a wildcard pattern.

Parameters:

startingDirName - - Directory to start searching from

fileNames - - Wildcard pattern to search for

Returns:

- Pair of lists; the first element is the list of local file names and the second a list of remote file names

Throws:

IOException

localDirExists

public static boolean localDirExists(String dirName)

This is an utility method that checks if a local directory exists or not

Parameters:

dirName - - Name of the directory

Returns:

- true if the directory exists and false if it doesn't exist or if the 'dirName' is null or empty.

localFileExists

public static boolean localFileExists(String fileName)

This method checks if a local file exists or not. If the file name is a wildcard pattern it skips the check.

Parameters:

fileName - - String - Name of the file

Returns:

- true if the file exists, false otherwise

getKifsPathSeparator

public static String getKifsPathSeparator()

parseFileNames

public static List<org.apache.commons.lang3.tuple.Triple<String,String,String>> parseFileNames(List<String> fileNames)

This method parses the list of file names passed in to the constructor of the classes FileUploader, FileDownloader which are derivatives of this class. It resolves the file names, normalizes them and returns a corresponding list of absolute paths which are used ny the methods of this class.

Parameters:

fileNames - - List of file names passed in to the constructor of the derivatives of this class.

Returns:

- Returns a list of Triple objects where the first element is the root of the file path, the second the full path without the file name and the third just the file name itself.