Skip to end of metadata
Go to start of metadata


Introduction

The selection & handling of the objects being transferred can be divided into a number of areas:

  • File selection - which broadly covers the specification of features that are related to the source side of the file transfer operation such as filtering, polling and directives.
  • File handling - which broadly covers features that are implemented on the target side of the operation such as atomic transfer and the compression, checking and appending of files.
  • Renaming - which is also related to the source side of the file transfer operation and - in contrast with the file selection and file handling features - is protocol-specific.
  • Transfer target specification - specification of the transfer target directory and the conditions under which files are to be written to it.

YADE can also be configured to check whether file transfer has been successfully completed. See:

  • the User Manual FileTransferOptions article which contains a description of how the Transactional parameter used to ensure that transfer of a group of files has been completed.
  • the Atomicity Parameter Reference article for a description of the AtomicPrefix and AtomicSuffix parameters which are used to ensure that transfer of individual files has been completed..

File Selection

How to specify file selection

File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for YADE: Copy, Move etc.

File selection is a source side operation and the YADE file transfer options are all specified as child elements of the SourceFileOptions element.

This configuration hierarchy is shown for the Copy operation as follows:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
          • SourceFileOptions
            • Selection (required)
            • optional file selection parameters
        • CopyTarget
          • copy target parameters
      • Move
      • etc.

Note that all the optional file selection parameters apply to all files matching the selection criteria.

File Selection Functions

YADE comes with a comprehensive set of file selection functions:

  • Select an individual file or folder: 

    • FilePathSelection
      • This parameter is used to specify an individual file or a folder from which all files will be processed.
  • Select a dynamic number of files from a folder:

    • FileSpecSelection
      • Selects all files from a folder that match a regular expression
      • Can be used to select files recursively
  • Select files from a list:

    • FileListSelection
      • This parameter is used to specify a number of files for processing.

These functions are specified using one of the child elements of the Selection element.

The element hierarchy for the Selection element when a Copy operation is specified is:

  • Profile 
    • Operation 
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions (required)
              • Selection (required)
                • FilePathSelection
                  • FilePath (required)
                  • Directory
                • FileSpecSelection
                  • FileSpec (required)
                  • Directory
                  • Recursive
                • FileListSelection
                  • FileList (required)
                  • Directory

Exact definitions for each file selection element and the interaction between elements can be found in the Selection Parameter Reference.

Checking files for availability before transfer (CheckSteadyState)

The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source, for example, when waiting for the receipt of large files before processing.

The file size is measured at regular intervals and until a steady state is determined.

Note that this method of determining whether a file is fully available is not 100% reliable. This is discussed in detail in the CheckSteadyState parameter reference along with a description of the three child elements of the CheckSteadyState parameter: CheckSteadyStateIntervalCheckSteadyStateCount and CheckSteadyStateErrorState.

The CheckSteadyState parameter is a child element of SourceFileOptions as shown: 

  • Profile  
    • Operation  
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions (required)
              • Selection (required)
              • CheckSteadyState (option)
                • CheckSteadyStateInterval
                • CheckSteadyStateCount
                • CheckSteadyStateErrorState

Behavior when transfer criteria are not met (Directives)

Directives can be seen as optional parameters that specify how YADE is to generally react at the file transfer source.

The Directives element itself is a child of the SourceFileOptions elements:

  • SourceFileOptions
    • Selection (required)
    • Directives (optional)
      • DisableErrorOnNoFilesFound

      • TransferZeroByteFiles

    • Other optional source file elements

The directives are:

  • DisableErrorOnNoFilesFound

    • The DisableErrorOnNoFilesFound parameter allows the default behavior of YADE to be changed.
      Note that the use of this parameter can modify the behavior of YADE when TransferZeroByteFiles is set.

  • TransferZeroByteFiles

    • The TransferZeroByteFiles parameter specifies how YADE is to handle zero byte files - i.e. whether they are to be transferred and processed by subsequent commands.
      See the Parameter Reference for a details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.

Polling for files

YADE can use Polling to wait for incoming files. The files to be polled are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.

Polling only continues until the polling criteria have been met (i.e. one or a minimum number of files have been found).

File transfer is started once polling has stopped.

When YADE is run as a job in the JobScheduler a job node can be specified for execution in case polling is unsuccessful.

A detailed list of the polling parameters that can be set and their definitions can be found in the Polling section of the Parameter Reference.

The Polling parameter is a child element of SourceFileOptions as shown: 

  • Profile  
    • Operation  
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions (required)
              • Selection (required)
              • CheckSteadyState (option)
              • Polling (option)
                • PollInterval
                • PollTimeout
                • etc
              • Other optional source file elements

Restricting the files to be transferred (multiple parameters)

The following file filtering options can be used to filter the files that are to be transferred:

  • FileAge

    • FileAge sets the minimum and/or maximum file age
  • FileSize

    • FileSize sets the minimum and/or maximum file size
  • ResultSet

    • The ResultsSet parameter handles the result set of a Selection of files from the source host. The result set can be written into a file or, when YADE is run as a JobScheduler job, checked against an expectation.

  • SkipFiles

    • SkipFiles allows the first and/or last files found to be omitted from transfer.
  • MaxFiles

    • The MaxFiles parameter allows the maximum number of files to be transferred to be set. All other files will be ignored.

A detailed description of the file filtering elements listed above can be found in the SourceFileOptions Parameter Reference.

This configuration hierarchy relevant for the SourceFileOptions parameters where a Copy operation is specified is:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions
              • Selection (required)
              • FileAge
              • FileSize
              • ResultSet
              • SkipFiles
              • MaxFiles
              • Other optional file selection elements

File Handling

File handling is used here to refer to aspects of the transfer that are carried out target side of a transfer operation.

How to specify file handling

File handling is specified in the Profiles branch of the configuration and can be applied to those Operation elements that can be defined for YADE: Copy, Move etc.

File handling is a target side operation, and the YADE file handling options are all specified as child elements of the TargetFileOptions element.

This configuration hierarchy is shown for the Copy operation as follows:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
          • SourceFileOptions
            • Selection (required)
            • optional file selection parameters
        • CopyTarget
          • CopyTargetFragmentRef (required)
          • Directory (required)
          • TargetFileOptions (option)
            • AppendFiles (option)
            • Atomicity (option)

            • etc.
      • Move
      • etc.

Note that all the optional file selection parameters apply to all files matching the selection criteria.

The file handling options

Appending one file to another (AppendFiles)

The AppendFiles element can be used to specify whether files are to be appended to other files (i.e. added to the end of a file already existing in the target directory)

AppendFiles is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically in the How to specify file handling section above:

Files are appended to files with the same name.

Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles (another optional child of TargetFileOptions) will be ignored.

Functioning

If we have two files to be transferred: file_a and file_b and a file_a already exists in the target directory, then the contents of the file_a being transferred will be appended to the already existing file_a. file_b will be written to the target directory as usual.

Atomicity

The Atomicity parameter is used to mask the presence of a file being transferred to a directory that may be monitored until the file has been fully transferred.

See the parameter reference for a detailed description of this parameter and its interaction with other elements.

Ensure that file transfer is complete:

  • CheckSize

    When specified, the CheckSize parameter causes the size (i.e. the number of bytes) of the files in the source and target directories to be compared.

    This method is not as accurate as CheckIntegrityHash.

  • CheckIntegrityHash

    The CheckIntegrityHash element causes integrity hash sums to be calculated for the files at source and target after transfer. If the hashes are not equal then the transfer will be rolled back.

    This method is more accurate than the CheckSize method.

Compress files (CompressFiles)

YADE can compress individual files before writing them to the target directory.

The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.

Note: YADE cannot combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).

Configure error handling (DisableErrorOnNoFilesFound)

The default configuration for YADE is that an error will be logged if a transfer should take place but no files are found.

This behavior can be altered with the DisableErrorOnNoFilesFound element, which is a child of the Directives element and which is itself a child of the SourceFileOptions elements.

See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.

CumulateFiles

The CumulateFiles element specifies whether the contents of the transferred file(s) should be appended to the contents of an already existing file. All files being transferred will be written to the one file.

Compress files at the target (CompressFiles)

The CompressFiles element causes files to be compressed after transfer but before being written to the target file system.

Files are compressed individually, the possibility of adding files to a compressed archive either before or after transmission is not available.

KeepModificationDate

The KeepModificationDate element specifies whether the modification date of the source file is to be applied to the target.

This option is protocol-dependent and requires that the necessary permissions are available.

See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with different protocols.

DisableMakeDirectories

The DisableMakeDirectories element specifies whether YADE is to create a target directory if the directory specified in the Directory parameter is not available.

This option is protocol-dependent and requires that the necessary permissions are available.

See the TargetFileOptions Parameter Reference for more detailed information about this parameter and its functioning with the Recursive parameter and different protocols.

DisableOverwriteFiles

The DisableOverwriteFiles element specifies whether files can be overwritten.

File Renaming

File renaming is specified using the Rename element, and can be applied at either the source or target sides of a file transfer.

Renaming can be applied to operations that are carried out using all protocols YADE can currently use (Release 1.11.x). 

File renaming is protocol-specific and is specified as a child element of FragmentRef elements such as FTPFragmentRef, SMBFragmentRef and LocalSource and LocalTarget elements in Profiles.

The element hierarchy related to a Rename parameter specified for a copy to a local target operation is:

  • Copy
    • CopySource
      • child elements ...
    • CopyTarget
      • CopyTargetFragmentRef
        • LocalTarget
          • Rename (optional)
            • ReplaceWhat
            • ReplaceWith

The Rename element requires that two child elements are defined:

  • ReplaceWhat - a regular expression that allows selection of parts of file names.
  • ReplaceWith - complex replacement patterns using substitution masks and capturing groups can be used.

A detailed description of the possible file name replacement patterns can be found in the Rename element parameter reference.

See also Atomicity.

The Target Directory

The Directory Element

The Directory element defines the target folder for a file transfer and is required for the Copy and Move operations.

For a Copy operation, the Directory element is specified as a sibling element of the CopyTargetFragmentRef and TargetFileOptions elements as shown schematically below.

  • Copy
    • CopySource
      • Child elements
    • CopyTarget
      • CopyTargetFragmentRef (required)
      • Directory (required)
      • TargetFileOptions (optional)

Writing files

  • DisableMakeDirectories

  • DisableOverwriteFiles

These elements are children of the TargetFileOptions element and are described in the File Handling section above.


  • No labels
Write a comment…