Table of Contents |
---|
Introduction
The selection & handling of the objects being transferred can be divided into a number of areas:
- File selection - which broadly covers the specification of features that are related to the source side of the file transfer operation such as filtering, polling and directives.
- File handling - which broadly covers features that are implemented on the target side of the operation such as atomic transfer and the compression, checking and appending of files.
- Renaming - which is also related to the source side of the file transfer operation and - in contrast with the file selection and file handling features - is protocol-specific.
...
- .
- Transfer target specification - specification of the transfer target directory and the conditions under which files are to be written to it.
YADE can also be configured to check whether file transfer has been successfully completed. See:
- the User Manual FileTransferOptions article which contains a description of how the Transactional parameter used to ensure that transfer of a group of files has been completed.
- the Atomicity Parameter Reference article for a description of the AtomicPrefix and AtomicSuffix parameters which are used to ensure that transfer of individual files has been completed..
File Selection
How to specify file selection
File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for JADEYADE: Copy, Move, etc etc.
File selection is a source side operation , and the JADE YADE file transfer options are all specified as child elements of the SourceFileOptions element.
This configuration hierarchy is shown for the Copy operation as follows:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- optional file selection parameters
- CopyTarget
- copy target parameters
- CopySource
- Move
- etc.
- Copy
- Operation
Note that all the optional file selection parameters apply to all files matching the selection criteria.
File Selection Functions
JADE YADE comes with a comprehensive set of file selection functions:
Select an individual file or folder:
- FilePathSelection
- This parameter is used to specify an individual file or a folder from which all files will be processed.
- FilePathSelection
Select a dynamic number of files from a folder:
- FileSpecSelection
- Selects all files from a folder that match a regular expression
- Can be used to select files recursively
- FileSpecSelection
Select files from a list:
- FileListSelection
- This parameter is used to specify a number of files for processing.
- FileListSelection
...
The element hierarchy for the Selection element when a Copy operation is specified is:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- FilePathSelection
- FilePath (required)
- Directory
- Recursive (availability starting with JS1 1.13.18)
- FileSpecSelection
- FileSpec (required)
- Directory
- ExcludedDirectories (availability starting with JS1 1.13.18)
- Recursive
- FileListSelection
- FileList (required)
- Directory
- Recursive (availability starting with JS1 1.13.18)
- FilePathSelection
- Selection (required)
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Exact definitions for each file selection element and the interaction between elements can be found in the Selection Parameter Parameter Reference.
Checking files for
...
availability before transfer (CheckSteadyState)
The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source, for example, when waiting for the receipt of large files before processing.
The file size is measured at regular intervals and until a steady state is determined.
Note that this method of determining whether a file is fully available is not 100% reliable. This is discussed in detail in the CheckSteadyState parameter reference alongside alternative approaches for avoiding corrupt files and along with a description of the three child elements of the CheckSteadyState parameter: CheckSteadyStateInterval, CheckSteadyStateCount and CheckSteadyStateErrorState.
The CheckSteadyState parameter is a child element of SourceFileOptions as shown:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- CheckSteadyState (option)
- CheckSteadyStateInterval
- CheckSteadyStateCount
- CheckSteadyStateErrorState
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Behavior when transfer criteria are not met (Directives)
...
...
Directives can be seen as optional parameters that specify how JADE YADE is to generally react at the file transfer source.
The Directives element itself is a child of the SourceFileOptions elements:
- SourceFileOptions
- Selection (required)
- Directives (optional)
DisableErrorOnNoFilesFound
TransferZeroByteFiles
- Other optional source file elements
The directives are:
DisableErrorOnNoFilesFound
...
The DisableErrorOnNoFilesFound parameter allows the default behavior of
...
YADE to be changed.
Note that the use of this parameter can modify the behavior of
...
YADE when TransferZeroByteFiles is set.
TransferZeroByteFiles
...
The TransferZeroByteFiles parameter specifies how
...
YADE is to handle zero byte files - i.e. whether they are to be transferred and processed by subsequent commands.
See the Parameter Reference for a details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.
Polling for files
JADE YADE can use polling Polling to wait for incoming files. The files to be polled are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.
...
File transfer is started once polling has stopped.
When JADE YADE is run as a job in the JobScheduler a job node can be specified for execution in case polling is unsuccessful.
A detailed list of the polling parameters that can be set and their definitions can be found in the Polling section of the Parameter Reference.
...
The Polling parameter is a child element of SourceFileOptions as shown:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- CheckSteadyState (option)
- Polling (option)
- PollInterval
- PollTimeout
- etc
- PollInterval
- Other optional source file elements
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Restricting the files to be transferred (multiple parameters)
The following file filtering options can be used to filter the files that are to be transferred:
ResultSet
The ResultsSet parameter
FileAge
- Sets the minimum and/or maximum file age
FileSize
- Sets the minimum and/or maximum file size
ResultSet
This handles the result set of a Selection of files from the source host. The result set can be written into a file or, when JADE YADE is run as a JobScheduler job, checked against an expectation.
SkipFiles
- Allows the first and/or last files found to be omitted from transfer.
MaxFiles
- The MaxFiles parameter allows
MaxFiles
- Allows the maximum number of files to be transferred to be set. All other files will be ignored.
- Note that it is not possible to specify which files will be ignored should the value set for this parameter be exceeded. It therefore only makes sense to use this parameter in particular situations such as when the contents of a directory will be repeatedly polled.
Status colour Yellow title Info to Param Ref
A detailed description of the file filtering elements listed above can be found in the SourceFileOptions Parameter Reference. Parameter Reference.
This configuration hierarchy relevant for the SourceFileOptions parameters where a Copy operation is specified is:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- ResultSet
- MaxFiles
- Other optional file selection elements
- SourceFileOptions
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
File Handling
File handling is used here to refer to operations aspects of the transfer that are carried out target side of a transfer operation.
How to specify file handling
File handling is specified in the in the Profiles branch of the configuration and can be applied to for to those Operations Operation elements that can be defined for JADEYADE: Copy, Move, etc etc.
File selection handling is a source target side operation, and the JADE YADE file transfer handling options are all specified as child elements of the SourceFileOptions TargetFileOptions element.
This configuration hierarchy is shown for the Copy operation as follows:
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- optional file selection parameters
- CopyTarget
- CopyTargetFragmentRef (required)
- Directory (required)
- TargetFileOptions (option)
- AppendFiles (option)
Atomicity (option)
- etc.
- copy target parameters
- CopySource
- Move
- etc.
- Copy
- Operation
Note that all the optional file selection parameters apply to all files matching the selection criteria.
...
The file handling options
Appending one file to another (AppendFiles)
The AppendFiles element can be used to specify that whether files are to be appended to other files (i.e. added to the end of a file already existing on in the target directory)
The AppendFiles element is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically below:
- Profile
- Operation
- Copy
- CopySource
- Child elements ...
- CopyTarget
- Child elements ...
- TransferOptions
- CopySource
- Move
- MoveSource
- Child elements ...
- MoveTarget
- Child elements ...
- TransferOptions
- MoveSource
- Copy
- Operation
in the How to specify file handling section above:
Files are appended to files with the same name. Status
Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles (another optional child of TargetFileOptions) will be ignored.
How to compress files
.
Functioning
If we have two files to be transferred: file_a and file_b and a file_a already exists in the target directory, then the contents of the file_a being transferred will be appended to the already existing file_a. file_b will be written to the target directory as usual.
Atomicity
The Atomicity parameter is used to mask the presence of a file being transferred to a directory that may be monitored until the file has been fully transferred.
See the parameter reference for a detailed description of this parameter and its interaction with other elements.
Ensure that file transfer is complete:
CheckSize
When specified, the CheckSize parameter causes the size (i.e. the number of bytes) of the files in the source and target directories to be compared.
This method is not as accurate as CheckIntegrityHash.
CheckIntegrityHash
The CheckIntegrityHash element causes integrity hash sums to be calculated for the files at source and target after transfer. If the hashes are not equal then the transfer will be rolled back.
This method is more accurate than the CheckSize method.
Compress files (CompressFiles)
YADE JADE can compress individual files before writing them to the target directory. Status
The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.
The CompressFiles element is a child element of the TargetFileOptions element, which itself can be specified for CopyTarget and MoveTarget elements in a Profile as shown schematically below.
- Profile
- Operation
- Copy
- CopyTarget
- TargetFileOptions
- CompressFiles
- TargetFileOptions
- etc ...
- CopyTarget
- Move
- MoveTarget
- TargetFileOptions
- CompressFiles
- TargetFileOptions
- etc ...
- MoveTarget
- Copy
- Operation
JADE is not yet able to Note: YADE cannot combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).Could this be done using SFTP and preprocessing? Status
How to handle Zero Byte Files
Status | ||||
---|---|---|---|---|
|
The handling of zero byte files is specified with the TransferZeroByteFiles element.
This element specifies whether zero byte files are to be transferred and processed by subsequent commands.
The options available and the interaction with the Remove and DisableErrorOnNoFilesFound elements are described in detail in the TransferZeroByteFiles parameter reference article.
The TransferZeroByteFiles element is a child of the Directives element, which itself is a child of the SourceFileOptions elements.
These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.
How to configure error handling
Configure error handling (DisableErrorOnNoFilesFound)
The default configuration for JADE YADE is that an error will be logged if a transfer should take place but no files are found.
This behavior can be altered with the DisableErrorOnNoFilesFound element, which is a child of the Directives element , and which is itself is a child of the SourceFileOptions elements.These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.
See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.
CumulateFiles
The CumulateFiles element specifies whether the contents of the transferred file(s) should be appended to the contents of an already existing file. All files being transferred will be written to the one file.
Compress files at the target (CompressFiles)
The CompressFiles element causes files to be compressed after transfer but before being written to the target file system.
Files are compressed individually, the possibility of adding files to a compressed archive either before or after transmission is not available.
KeepModificationDate
The KeepModificationDate element specifies whether the modification date of the source file is to be applied to the target.
This option is protocol-dependent and requires that the necessary permissions are available.
See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with different protocols.
DisableMakeDirectories
The DisableMakeDirectories element specifies whether YADE is to create a target directory if the directory specified in the Directory parameter is not available.
This option is protocol-dependent and requires that the necessary permissions are available.
See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with the Recursive parameter and different protocols.
DisableOverwriteFiles
The DisableOverwriteFiles element specifies whether files can be overwritten.
File Renaming
File renaming is specified using the Rename element, and can be applied at either the source or target sides of a file transfer.
Renaming can be applied to operations that are carried out using all protocols JADE YADE can currently use (Release 1.11.x) use.
File renaming is protocol-specific and is specified as a child element of FragmentRef elements such as FTPFragkentRef FTPFragmentRef, SMBFragmentRef and LocalSource and LocalTarget elements in Profiles.
The element hierarchy related to a Rename parameter specified for a copy to a local target operation is:
...
A detailed description of the possible file name replacement patterns can be found in the Rename element element parameter reference.
See also Atomicity.
File Transfer Options
File transfer options are all optional and are used to:
- optimize file transfer (buffer size, concurrent transfer) and
- handle errors (transactional transfer)
Specification of File Transfer Options
File Transfer Options are defined using the FileTransferOptions elements. FileTransferOptions elements are specified as part of a file transfer Profile, as child elements of the Copy or Move elements as shown in the following diagram:
...
The Target Directory
The Directory Element
The Directory element defines the target folder for a file transfer and is required for the Copy and Move operations.
For a Copy operation, the Directory element is specified as a sibling element of the CopyTargetFragmentRef and TargetFileOptions elements as shown schematically below.
...
- Copy
- CopySource
- Child elements ...
- CopyTarget
- Child elements ...
- TransferOptions
- Move
- MoveSource
- Child elements ...
- MoveTarget
- Child elements ...
- TransferOptions
- MoveSource
- CopySource
The JADE file transfer options are independent of the transfer protocol. As they apply to the transfer, it follows that they are independent of the source /target and are specified as sibling elements of - in the cases of a Move operation - the MoveSource and MoveTarget elements.
The TransferOptions element can have three children as listed below. The use of these child elements is described in the next sections:
- TransferOptions
- BufferSize
- ConcurrentTransfer
- MaxConcurrentTransfers
- Transactional
Transfer Optimization 1 - Buffer Size
The BufferSize parameter element defines the maximum data block size.
Status | ||||
---|---|---|---|---|
|
Transfer Optimization 2 - Concurrent Transfer
The ConcurrentTransfer element is used to allow parallel file transfer to occur.
A child element of ConcurrentTransfer - MaxConcurrentTransfers - can be used to limit the number of parallel transfers.
- TransferOptions
- ConcurrentTransfer
- MaxConcurrentTransfers
- ConcurrentTransfer
Transactional Transfer
This parameter specifies whether a transfer should be processed as a single transaction, i.e. whether all objects are to be successfully transferred or none at all.
Should an error occur during a transfer operation then transfers already carried out will be rolled back.
Transactional transfer is often used together with the following parameters:
- Atomicity (a TargetFileOption for the CopyTarget and MoveTarget elements)
- DisableOverwriteFiles (a TargetFileOption for the CopyTarget and MoveTarget elements)
See the Transactional Parameter Reference Article for a detailed explanation of the dependencies between these elements.
Note that transactional transfer is not 100% reliable - there are situations where rollback will not be possible.
- CopyTargetFragmentRef (required)
- Directory (required)
- TargetFileOptions (optional)
Writing files
DisableMakeDirectories
DisableOverwriteFiles
These elements are children of the TargetFileOptions element and are described in the File Handling section above.