Introduction
The selection & handling of the objects being transferred can be divided into a number of areas:
- File selection - which broadly covers the specification of features that are related to the source side of the file transfer operation such as filtering, polling and directives.
- File handling - which broadly covers features that are implemented on the target side of the operation such as atomic transfer and the compression, checking and appending of files.
- Renaming - which is also related to the source side of the file transfer operation and - in contrast with the file selection and file handling features - is protocol-specific.
- Transfer target specification - specification of the transfer target directory and the conditions under which files are to be written to it.
YADE can also be configured to check whether file transfer has been successfully completed. See:
- the User Manual FileTransferOptions article which contains a description of how the Transactional parameter used to ensure that transfer of a group of files has been completed.
- the Atomicity Parameter Reference article for a description of the AtomicPrefix and AtomicSuffix parameters which are used to ensure that transfer of individual files has been completed..
File Selection
How to specify file selection
File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for YADE: Copy, Move etc.
File selection is a source side operation and the YADE file transfer options are all specified as child elements of the SourceFileOptions element.
This configuration hierarchy is shown for the Copy operation as follows:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- optional file selection parameters
- CopyTarget
- copy target parameters
- CopySource
- Move
- etc.
- Copy
- Operation
Note that all the optional file selection parameters apply to all files matching the selection criteria.
File Selection Functions
YADE comes with a comprehensive set of file selection functions:
Select an individual file or folder:
- FilePathSelection
- This parameter is used to specify an individual file or a folder from which all files will be processed.
- FilePathSelection
Select a dynamic number of files from a folder:
- FileSpecSelection
- Selects all files from a folder that match a regular expression
- Can be used to select files recursively
- FileSpecSelection
Select files from a list:
- FileListSelection
- This parameter is used to specify a number of files for processing.
- FileListSelection
These functions are specified using one of the child elements of the Selection element.
The element hierarchy for the Selection element when a Copy operation is specified is:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- FilePathSelection
- FilePath (required)
- Directory
- Recursive (availability starting with JS1 1.13.18)
- FileSpecSelection
- FileSpec (required)
- Directory
- ExcludedDirectories (availability starting with JS1 1.13.18)
- Recursive
- FileListSelection
- FileList (required)
- Directory
- Recursive (availability starting with JS1 1.13.18)
- FilePathSelection
- Selection (required)
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Exact definitions for each file selection element and the interaction between elements can be found in the Selection Parameter Reference.
Checking files for availability before transfer (CheckSteadyState)
The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source, for example, when waiting for the receipt of large files before processing.
The file size is measured at regular intervals and until a steady state is determined.
Note that this method of determining whether a file is fully available is not 100% reliable. This is discussed in detail in the CheckSteadyState parameter reference along with a description of the three child elements of the CheckSteadyState parameter: CheckSteadyStateInterval, CheckSteadyStateCount and CheckSteadyStateErrorState.
The CheckSteadyState parameter is a child element of SourceFileOptions as shown:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- CheckSteadyState (option)
- CheckSteadyStateInterval
- CheckSteadyStateCount
- CheckSteadyStateErrorState
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Behavior when transfer criteria are not met (Directives)
Directives can be seen as optional parameters that specify how YADE is to generally react at the file transfer source.
The Directives element itself is a child of the SourceFileOptions elements:
- SourceFileOptions
- Selection (required)
- Directives (optional)
DisableErrorOnNoFilesFound
TransferZeroByteFiles
- Other optional source file elements
The directives are:
DisableErrorOnNoFilesFound
The DisableErrorOnNoFilesFound parameter allows the default behavior of YADE to be changed.
Note that the use of this parameter can modify the behavior of YADE when TransferZeroByteFiles is set.
TransferZeroByteFiles
The TransferZeroByteFiles parameter specifies how YADE is to handle zero byte files - i.e. whether they are to be transferred and processed by subsequent commands.
See the Parameter Reference for a details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.
Polling for files
YADE can use Polling to wait for incoming files. The files to be polled are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.
Polling only continues until the polling criteria have been met (i.e. one or a minimum number of files have been found).
File transfer is started once polling has stopped.
When YADE is run as a job in the JobScheduler a job node can be specified for execution in case polling is unsuccessful.
A detailed list of the polling parameters that can be set and their definitions can be found in the Polling section of the Parameter Reference.
The Polling parameter is a child element of SourceFileOptions as shown:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- CheckSteadyState (option)
- Polling (option)
- PollInterval
- PollTimeout
- etc
- PollInterval
- Other optional source file elements
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Restricting the files to be transferred (multiple parameters)
The following file filtering options can be used to filter the files that are to be transferred:
ResultSet
The ResultsSet parameter handles the result set of a Selection of files from the source host. The result set can be written into a file or, when YADE is run as a JobScheduler job, checked against an expectation.
MaxFiles
- The MaxFiles parameter allows the maximum number of files to be transferred to be set. All other files will be ignored.
A detailed description of the file filtering elements listed above can be found in the SourceFileOptions Parameter Reference.
This configuration hierarchy relevant for the SourceFileOptions parameters where a Copy operation is specified is:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- ResultSet
- MaxFiles
- Other optional file selection elements
- SourceFileOptions
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
File Handling
File handling is used here to refer to aspects of the transfer that are carried out target side of a transfer operation.
How to specify file handling
File handling is specified in the Profiles branch of the configuration and can be applied to those Operation elements that can be defined for YADE: Copy, Move etc.
File handling is a target side operation, and the YADE file handling options are all specified as child elements of the TargetFileOptions element.
This configuration hierarchy is shown for the Copy operation as follows:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- optional file selection parameters
- CopyTarget
- CopyTargetFragmentRef (required)
- Directory (required)
- TargetFileOptions (option)
- AppendFiles (option)
Atomicity (option)
- etc.
- CopySource
- Move
- etc.
- Copy
- Operation
Note that all the optional file selection parameters apply to all files matching the selection criteria.
The file handling options
Appending one file to another (AppendFiles)
The AppendFiles element can be used to specify whether files are to be appended to other files (i.e. added to the end of a file already existing in the target directory)
AppendFiles is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically in the How to specify file handling section above:
Files are appended to files with the same name.
Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles (another optional child of TargetFileOptions) will be ignored.
Functioning
If we have two files to be transferred: file_a and file_b and a file_a already exists in the target directory, then the contents of the file_a being transferred will be appended to the already existing file_a. file_b will be written to the target directory as usual.
Atomicity
The Atomicity parameter is used to mask the presence of a file being transferred to a directory that may be monitored until the file has been fully transferred.
See the parameter reference for a detailed description of this parameter and its interaction with other elements.
Ensure that file transfer is complete:
CheckSize
When specified, the CheckSize parameter causes the size (i.e. the number of bytes) of the files in the source and target directories to be compared.
This method is not as accurate as CheckIntegrityHash.
CheckIntegrityHash
The CheckIntegrityHash element causes integrity hash sums to be calculated for the files at source and target after transfer. If the hashes are not equal then the transfer will be rolled back.
This method is more accurate than the CheckSize method.
Compress files (CompressFiles)
YADE can compress individual files before writing them to the target directory.
The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.
Note: YADE cannot combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).
Configure error handling (DisableErrorOnNoFilesFound)
The default configuration for YADE is that an error will be logged if a transfer should take place but no files are found.
This behavior can be altered with the DisableErrorOnNoFilesFound element, which is a child of the Directives element and which is itself a child of the SourceFileOptions elements.
See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.
CumulateFiles
The CumulateFiles element specifies whether the contents of the transferred file(s) should be appended to the contents of an already existing file. All files being transferred will be written to the one file.
Compress files at the target (CompressFiles)
The CompressFiles element causes files to be compressed after transfer but before being written to the target file system.
Files are compressed individually, the possibility of adding files to a compressed archive either before or after transmission is not available.
KeepModificationDate
The KeepModificationDate element specifies whether the modification date of the source file is to be applied to the target.
This option is protocol-dependent and requires that the necessary permissions are available.
See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with different protocols.
DisableMakeDirectories
The DisableMakeDirectories element specifies whether YADE is to create a target directory if the directory specified in the Directory parameter is not available.
This option is protocol-dependent and requires that the necessary permissions are available.
See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with the Recursive parameter and different protocols.
DisableOverwriteFiles
The DisableOverwriteFiles element specifies whether files can be overwritten.
File Renaming
File renaming is specified using the Rename element, and can be applied at either the source or target sides of a file transfer.
Renaming can be applied to operations that are carried out using all protocols YADE can currently use (Release 1.11.x).
File renaming is protocol-specific and is specified as a child element of FragmentRef elements such as FTPFragmentRef, SMBFragmentRef and LocalSource and LocalTarget elements in Profiles.
The element hierarchy related to a Rename parameter specified for a copy to a local target operation is:
- Copy
- CopySource
- child elements ...
- CopyTarget
- CopyTargetFragmentRef
- LocalTarget
- Rename (optional)
- ReplaceWhat
- ReplaceWith
- Rename (optional)
- LocalTarget
- CopyTargetFragmentRef
- CopySource
The Rename element requires that two child elements are defined:
- ReplaceWhat - a regular expression that allows selection of parts of file names.
- ReplaceWith - complex replacement patterns using substitution masks and capturing groups can be used.
A detailed description of the possible file name replacement patterns can be found in the Rename element parameter reference.
See also Atomicity.
The Target Directory
The Directory Element
The Directory element defines the target folder for a file transfer and is required for the Copy and Move operations.
For a Copy operation, the Directory element is specified as a sibling element of the CopyTargetFragmentRef and TargetFileOptions elements as shown schematically below.
- Copy
- CopySource
- Child elements
- CopyTarget
- CopyTargetFragmentRef (required)
- Directory (required)
- TargetFileOptions (optional)
- CopySource
Writing files
DisableMakeDirectories
DisableOverwriteFiles
These elements are children of the TargetFileOptions element and are described in the File Handling section above.