Show If | |||||
---|---|---|---|---|---|
| |||||
|
Table of Contents |
---|
Introduction
The selection & handling of the objects being transferred can be divided into a number of areas:
...
- the User Manual FileTransferOptions article which contains a description of the Transactional parameter used to ensure that transfer of a group of files has been completed.
- the Atomicity Parameter Reference article for a description of the AtomicPrefix and AtomicSuffix parameters which are used to ensure that transfer of individual files has been completed..
Configuring File Selection, Handling and Renaming with the XML Editor
Excerpt Include | ||||
---|---|---|---|---|
|
File Selection
How to specify file selection
File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for JADE: Copy, Move etc.
...
Note that all the optional file selection parameters apply to all files matching the selection criteria.
File Selection Functions
JADE comes with a comprehensive set of file selection functions:
Select an individual file or folder:
- FilePathSelection
- This parameter is used to specify an individual file or a folder from which all files will be processed.
- FilePathSelection
Select a dynamic number of files from a folder:
- FileSpecSelection
- Selects all files from a folder that match a regular expression
- Can be used to select files recursively
- FileSpecSelection
Select files from a list:
- FileListSelection
- This parameter is used to specify a number of files for processing.
- FileListSelection
...
Exact definitions for each file selection element and the interaction between elements can be found in the Selection Parameter Reference.
Checking files for availability before transfer (CheckSteadyState)
The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source, for example, when waiting for the receipt of large files before processing.
...
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- CheckSteadyState (option)
- CheckSteadyStateInterval
- CheckSteadyStateCount
- CheckSteadyStateErrorState
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Behavior when transfer criteria are not met (Directives)
Directives can be seen as optional parameters that specify how JADE is to generally react at the file transfer source.
...
- SourceFileOptions
- Selection (required)
- Directives (optional)
DisableErrorOnNoFilesFound
TransferZeroByteFiles
- Other optional source file elements
The directives are:
DisableErrorOnNoFilesFound
The DisableErrorOnNoFilesFound parameter allows the default behavior of JADE to be changed.
Note that the use of this parameter can modify the behavior of JADE when TransferZeroByteFiles is set.
TransferZeroByteFiles
The TransferZeroByteFiles parameter specifies how JADE is to handle zero byte files - i.e. whether they are to be transferred and processed by subsequent commands.
See the Parameter Reference for a details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.
Polling for files
JADE can use Polling to wait for incoming files. The files to be polled are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.
...
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions (required)
- Selection (required)
- CheckSteadyState (option)
- Polling (option)
- PollInterval
- PollTimeout
- etc
- PollInterval
- Other optional source file elements
- SourceFileOptions (required)
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
Restricting the files to be transferred (multiple parameters)
The following file filtering options can be used to filter the files that are to be transferred:
FileAge
- FileAge sets the minimum and/or maximum file age
FileSize
- FileSize sets the minimum and/or maximum file size
ResultSet
The ResultsSet parameter handles the result set of a Selection of files from the source host. The result set can be written into a file or, when JADE is run as a JobScheduler job, checked against an expectation.
SkipFiles
- SkipFiles allows the first and/or last files found to be omitted from transfer.
MaxFiles
- The MaxFiles parameter allows the maximum number of files to be transferred to be set. All other files will be ignored.
...
- Profile
- Operation
- Copy
- CopySource
- CopySourceFragmentRef
- SourceFileOptions
- Selection (required)
- FileAge
- FileSize
- ResultSet
- SkipFiles
- MaxFiles
- Other optional file selection elements
- SourceFileOptions
- CopySourceFragmentRef
- CopySource
- Copy
- Operation
File Handling
File handling is used here to refer to aspects of the transfer that are carried out target side of a transfer operation.
How to specify file handling
File handling is specified in the Profiles branch of the configuration and can be applied to those Operation elements that can be defined for JADE: Copy, Move etc.
...
Note that all the optional file selection parameters apply to all files matching the selection criteria.
The file handling options
Appending one file to another (AppendFiles)
The AppendFiles element can be used to specify whether files are to be appended to other files (i.e. added to the end of a file already existing in the target directory)
AppendFiles is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically in the JADE User Manual - File Selection, Handling and Renaming and the Target Directory2Directory section above:
Files are appended to files with the same name.
Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles (another optional child of TargetFileOptions) will be ignored.
Functioning
If we have two files to be transferred: file_a and file_b and a file_a already exists in the target directory, then the contents of the file_a being transferred will be appended to the already existing file_a. file_b will be written to the target directory as usual.
Atomicity
The Atomicity parameter is used to mask the presence of a file being transferred to a directory that may be monitored until the file has been fully transferred.
See the parameter reference for a detailed description of this parameter and its interaction with other elements.
Ensure that file transfer is complete:
CheckSize
When specified, the CheckSize parameter causes the size (i.e. the number of bytes) of the files in the source and target directories to be compared.
This method is not as accurate as CheckIntegrityHash.
CheckIntegrityHash
The CheckIntegrityHash element causes integrity hash sums to be calculated for the files at source and target after transfer. If the hashes are not equal then the transfer will be rolled back.
This method is more accurate than the JADE User Manual - File Selection, Handling and Renaming and the Target Directory2Directory method.
Compress files (CompressFiles)
JADE can compress individual files before writing them to the target directory.
...
Note: JADE cannot combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).
Configure error handling (DisableErrorOnNoFilesFound)
The default configuration for JADE is that an error will be logged if a transfer should take place but no files are found.
...
See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.
CumulateFiles
The CumulateFiles element specifies whether the contents of the transferred file(s) should be appended to the contents of an already existing file. All files being transferred will be written to the one file.
Compress files at the target (CompressFiles)
The CompressFiles element causes files to be compressed after transfer but before being written to the target file system.
Files are compressed individually, the possibility of adding files to a compressed archive either before or after transmission is not available.
KeepModificationDate
The KeepModificationDate element specifies whether the modification date of the source file is to be applied to the target.
...
See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with different protocols.
DisableMakeDirectories
The DisableMakeDirectories element specifies whether JADE is to create a target directory if the directory specified in the Directory parameter is not available.
...
See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with the Recursive parameter and different protocols.
DisableOverwriteFiles
The DisableOverwriteFiles element specifies whether files can be overwritten.
File Renaming
File renaming is specified using the Rename element, and can be applied at either the source or target sides of a file transfer.
...
A detailed description of the possible file name replacement patterns can be found in the Rename element parameter reference.
See also Atomicity.
The Target Directory
The Directory Element
The Directory element defines the target folder for a file transfer and is required for the Copy and Move operations.
...
- Copy
- CopySource
- Child elements
- CopyTarget
- CopyTargetFragmentRef (required)
- Directory (required)
- TargetFileOptions (optional)
- CopySource
Writing files
DisableMakeDirectories
DisableOverwriteFiles
These elements are children of the TargetFileOptions element and are described in the JADE User Manual - File Selection, Handling and Renaming and the Target Directory2Directory section above.