Page History

Introduction

The selection & handling of the objects being transferred can be divided into a number of areas:

• File selection - which broadly covers the specification of features that are related to the source side of the file transfer operation such as filtering, polling and directives.
• File handling - which broadly covers features that are implemented on the target side of the operation such as atomic transfer and the compression, checking and appending of files.
• Renaming - which is also related to the source side of the file transfer operation and - in contrast with the file selection and file handling features - is protocol-specific.
• Transfer target specification - specification of the transfer target directory and the conditions under which files are to be written to it.

JADE YADE can also be configured to check whether file transfer has been successfully completed. See:

• the User Manual FileTransferOptions article which contains a description of how the Transactional parameter used to ensure that transfer of a group of files has been completed.
• the Atomicity Parameter Reference article for a description of the AtomicPrefix and AtomicSuffix parameters which are used to ensure that transfer of individual files has been completed..

File Selection

How to specify file selection

File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for JADEYADE: Copy, Move etc.

File selection is a source side operation and the JADE YADE file transfer options are all specified as child elements of the SourceFileOptions element.

...

Note that all the optional file selection parameters apply to all files matching the selection criteria.

File Selection Functions

JADE YADE comes with a comprehensive set of file selection functions:

• Select an individual file or folder: 

  • FilePathSelection
    • This parameter is used to specify an individual file or a folder from which all files will be processed.

• Select a dynamic number of files from a folder:

  • FileSpecSelection
    
    • Selects all files from a folder that match a regular expression
    • Can be used to select files recursively

• Select files from a list:

  • FileListSelection
    • This parameter is used to specify a number of files for processing.

...

Exact definitions for each file selection element and the interaction between elements can be found in the Selection Parameter Reference.

Checking files for availability before transfer (CheckSteadyState)

The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source, for example, when waiting for the receipt of large files before processing.

...

• Profile  
  • Operation  
    • Copy
      • CopySource
        • CopySourceFragmentRef
          • SourceFileOptions (required)
            • Selection (required)
            • CheckSteadyState (option)
              • CheckSteadyStateInterval
              • CheckSteadyStateCount
              • CheckSteadyStateErrorState

Behavior when transfer criteria are not met (Directives)

Directives can be seen as optional parameters that specify how JADE YADE is to generally react at the file transfer source.

...

• SourceFileOptions
  • Selection (required)
  • Directives (optional)

    • DisableErrorOnNoFilesFound

    • TransferZeroByteFiles

  • Other optional source file elements

The directives are:

• DisableErrorOnNoFilesFound

  • The DisableErrorOnNoFilesFound parameter allows the default behavior of JADE YADE to be changed.
    Note that the use of this parameter can modify the behavior of JADE YADE when TransferZeroByteFiles is set.

• TransferZeroByteFiles

  • The TransferZeroByteFiles parameter specifies how JADE YADE is to handle zero byte files - i.e. whether they are to be transferred and processed by subsequent commands.
    See the Parameter Reference for a details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.

Polling for files

JADE YADE can use Polling to wait for incoming files. The files to be polled are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.

...

File transfer is started once polling has stopped.

When JADE YADE is run as a job in the JobScheduler a job node can be specified for execution in case polling is unsuccessful.

...

• Profile  
  • Operation  
    • Copy
      • CopySource
        • CopySourceFragmentRef
          • SourceFileOptions (required)
            • Selection (required)
            • CheckSteadyState (option)
            • Polling (option)
              
              • PollInterval
                
              • PollTimeout
              • etc
            • Other optional source file elements

Restricting the files to be transferred (multiple parameters)

The following file filtering options can be used to filter the files that are to be transferred:

• FileAge

  • FileAge sets the minimum and/or maximum file age

• FileSize

  • FileSize sets the minimum and/or maximum file size

• ResultSet

  • The ResultsSet parameter handles the result set of a Selection of files from the source host. The result set can be written into a file or, when JADE YADE is run as a JobScheduler job, checked against an expectation.

• SkipFiles

  • SkipFiles allows the first and/or last files found to be omitted from transfer.

• MaxFiles

  • The MaxFiles parameter allows the maximum number of files to be transferred to be set. All other files will be ignored.

...

• Profile
  • Operation
    • Copy
      • CopySource
        • CopySourceFragmentRef
          • SourceFileOptions
            • Selection (required)
            • FileAge
            • FileSize
            • ResultSet
            • SkipFiles
            • MaxFiles
            • Other optional file selection elements

File Handling

File handling is used here to refer to aspects of the transfer that are carried out target side of a transfer operation.

How to specify file handling

File handling is specified in the Profiles branch of the configuration and can be applied to those Operation elements that can be defined for JADEYADE: Copy, Move etc.

File handling is a target side operation, and the JADE YADE file handling options are all specified as child elements of the TargetFileOptions element.

...

Note that all the optional file selection parameters apply to all files matching the selection criteria.

The file handling options

Appending one file to another (AppendFiles)

The AppendFiles element can be used to specify whether files are to be appended to other files (i.e. added to the end of a file already existing in the target directory)

AppendFiles is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically in the JADE User Manual - File Selection, Handling and Renaming and the Target Directory How to specify file handling section above:

Files are appended to files with the same name.

Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles (another optional child of TargetFileOptions) will be ignored.

Functioning

If we have two files to be transferred: file_a and file_b and a file_a already exists in the target directory, then the contents of the file_a being transferred will be appended to the already existing file_a. file_b will be written to the target directory as usual.

Atomicity

The Atomicity parameter is used to mask the presence of a file being transferred to a directory that may be monitored until the file has been fully transferred.

See the parameter reference for a detailed description of this parameter and its interaction with other elements.

Ensure that file transfer is complete:

• CheckSize

  When specified, the CheckSize parameter causes the size (i.e. the number of bytes) of the files in the source and target directories to be compared.

  This method is not as accurate as CheckIntegrityHash.

• CheckIntegrityHash

  The CheckIntegrityHash element causes integrity hash sums to be calculated for the files at source and target after transfer. If the hashes are not equal then the transfer will be rolled back.

  This method is more accurate than the JADE User Manual - File Selection, Handling and Renaming and the Target Directory CheckSize method.

Compress files (CompressFiles)

JADE YADE can compress individual files before writing them to the target directory.

The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.

Note: JADE YADE cannot combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).

Configure error handling (DisableErrorOnNoFilesFound)

The default configuration for JADE YADE is that an error will be logged if a transfer should take place but no files are found.

...

See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.

CumulateFiles

The CumulateFiles element specifies whether the contents of the transferred file(s) should be appended to the contents of an already existing file. All files being transferred will be written to the one file.

Compress files at the target (CompressFiles)

The CompressFiles element causes files to be compressed after transfer but before being written to the target file system.

Files are compressed individually, the possibility of adding files to a compressed archive either before or after transmission is not available.

KeepModificationDate

The KeepModificationDate element specifies whether the modification date of the source file is to be applied to the target.

...

See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with different protocols.

DisableMakeDirectories

The DisableMakeDirectories element specifies whether JADE YADE is to create a target directory if the directory specified in the Directory parameter is not available.

...

See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with the Recursive parameter and different protocols.

DisableOverwriteFiles

The DisableOverwriteFiles element specifies whether files can be overwritten.

File Renaming

File renaming is specified using the Rename element, and can be applied at either the source or target sides of a file transfer.

Renaming can be applied to operations that are carried out using all protocols JADE YADE can currently use (Release 1.11.x). 

...

A detailed description of the possible file name replacement patterns can be found in the Rename element parameter reference.

See also Atomicity.

The Target Directory

The Directory Element

The Directory element defines the target folder for a file transfer and is required for the Copy and Move operations.

...

• Copy
  • CopySource
    • Child elements
  • CopyTarget
    • CopyTargetFragmentRef (required)
    • Directory (required)
    • TargetFileOptions (optional)

Writing files

• DisableMakeDirectories

• DisableOverwriteFiles

These elements are children of the TargetFileOptions element and are described in the JADE User Manual - File Selection, Handling and Renaming and the Target Directory section above.