Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 28 Next »

 

Introduction

The selection & handling of the objects being transferred can be divided into a number of areas:

  • File selection - which broadly covers the specification of features that are related to the source side of the file transfer operation such as filtering, polling and directives.
  • File handling - which broadly covers features that are implemented on the target side of the operation such as atomic transfer and the compression, checking and appending of files.
  • Renaming - which is also related to the source side of the file transfer operation and is protocol-specific.

 

Checking files for completeness is handled in the next page.

File Selection

File selection is carried out

How to specify file selection

File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for JADE: Copy, Move, etc.

File selection is a source side operation, and the JADE file transfer options are all specified as child elements of the SourceFileOptions element.

This configuration hierarchy is shown for the Copy operation as follows:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
          • SourceFileOptions
            • Selection (required)
            • optional file selection parameters
        • CopyTarget
          • copy target parameters
      • Move
      • etc.

Note that all the optional file selection parameters apply to all files matching the selection criteria.

File Selection

JADE comes with a comprehensive set of file selection functions:

  • Select an individual file or folder: 
    • FilePathSelection
      • This parameter is used to specify an individual file or a folder from which all files will be processed.
  • Select a dynamic number of files from a folder:
    • FileSpecSelection
      • Selects all files from a folder that match a regular expression
      • Can be used to select files recursively
  • Select files from a list:
    • FileListSelection
      • This parameter is used to specify a number of files for processing.

These functions are specified using one of the child elements of the Selection element.

The element hierarchy for the Selection element is:

  • SourceFileOptions (required)
    • Selection (required)
      • FilePathSelection
        • FilePath (required)
        • Directory
      • FileSpecSelection
        • FileSpec (required)
        • Directory
        • Recursive
      • FileListSelection
        • FileList (required)
        • Directory

Exact definitions for each file selection element can be found in the Selection Parameter Reference.

Checking files for completeness (CheckSteadyState)

The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source.

The file size is measured at regular intervals and until a steady state is determined.

Note that this method of determining whether a file is fully available is not 100% reliable. This is discussed in detail in the CheckSteadyState parameter reference alongside alternative approaches for avoiding corrupt files and a description of the three child elements of the CheckSteadyState parameter: CheckSteadyStateInterval, CheckSteadyStateCount and CheckSteadyStateErrorState.

Behavior when transfer criteria are not met (Directives) ÜBERARBEITEN

Directives can be seen as parameters that specify how JADE is to generally react at the file transfer source.

The directives are:

DisableErrorOnNoFilesFound

This parameter allows the default behavior of JADE to be changed.

Note that the use of this parameter can modify the behavior of JADE when TransferZeroByteFiles is set.

TransferZeroByteFiles

This parameter specifies how JADE is to handle zero byte files.

See the Parameter Reference for details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.

Polling

JADE can use polling to wait for incoming files. The files are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.

Polling only continues until the polling criteria have been met (i.e. one or a minimum number of files have been found).

File transfer is started once polling has stopped.

When JADE is run as a job in the JobScheduler a job node can be specified for execution in case polling is unsuccessful.

A detailed list of the polling parameters that can be set and their definitions can be found in the Polling section of the Parameter Reference.

Restricting the Files to be transferred (multiple parameters)

The following file filtering options can be used to filter the files that are to be transferred:

  • FileAge

    • Sets the minimum and/or maximum file age

  • FileSize

    • Sets the minimum and/or maximum file size

  • ResultSet

    • This handles the result set of a Selection of files from the source host. The result set can be written into a file or, when JADE is run as a JobScheduler job, checked against an expectation.

  • SkipFiles

    • Allows the first and/or last files found to be omitted from transfer.

  • MaxFiles

    • Allows the maximum number of files to be transferred to be set. All other files will be ignored.
    • DETERMINED CHRONOL.?

File Handling

File handling

How to append one file to another

The AppendFiles element can be used to specify that files are appended to other files (i.e. added to the end of a file already existing on the target directory)

The AppendFiles element is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically below:

  • Profile
    • Operation
      • Copy
        • CopySource
          • Child elements ...
        • CopyTarget
          • Child elements ...
        • TransferOptions
      • Move
        • MoveSource
          • Child elements ...
        • MoveTarget
          • Child elements ...
        • TransferOptions

Files are appended to files with the same name. VERIFY!

Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles will be ignored.

How to compress files

JADE can compress individual files before writing them to the target directory. VERIFY!

The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.

The CompressFiles element is a child element of the TargetFileOptions element, which itself can be specified for CopyTarget and MoveTarget elements in a Profile as shown schematically below.

  • Profile
    • Operation
      • Copy
        • CopyTarget
          • TargetFileOptions
            • CompressFiles
        • etc ...
      • Move
        • MoveTarget
          • TargetFileOptions
            • CompressFiles
        • etc ...

JADE is not yet able to combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).

Could this be done using SFTP and preprocessing?VERIFY!

 

How to handle Zero Byte Files EVTL. MOVE TO SELECTION

The handling of zero byte files is specified with the TransferZeroByteFiles element.

This element specifies whether zero byte files are to be transferred and processed by subsequent commands.

The options available and the interaction with the Remove and DisableErrorOnNoFilesFound elements are described in detail in the TransferZeroByteFiles parameter reference article.

The TransferZeroByteFiles element is a child of the Directives element, which itself is a child of the SourceFileOptions elements.

These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.

How to configure error handling

The default configuration for JADE is that an error will be logged if a transfer should take place but no files are found

This behavior can altered with the DisableErrorOnNoFilesFound element, which is a child of the Directives element, which itself is a child of the SourceFileOptions elements.

These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.

See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.

File Transfer Options

Move Transactional?

Move to Error Handling section?

See also ErrorHandling (above)

File transfer options are used to:

  • optimize file transfer (buffer size, concurrent transfer) and
  • handle errors (transactional transfer)

Specification of File Transfer Options

File Transfer Options are defined using the FileTransferOptions elements. FileTransferOptions elements are specified as part of a file transfer Profile, as child elements of the Copy or Move elements as shown in the following diagram:

  • Profile
    • Operation
      • Copy
        • CopySource
          • Child elements ...
        • CopyTarget
          • Child elements ...
        • TransferOptions
      • Move
        • MoveSource
          • Child elements ...
        • MoveTarget
          • Child elements ...
        • TransferOptions

It follows that FileTransferOptions elements are independent of the file transfer fragment specified and that separate Options can be specified for the source and target parts of the transfer.

The TransferOptions elements can have three children as listed below. The use of these child elements is described in the next sections:

  • TransferOptions
    • BufferSize
    • ConcurrentTransfer
      • MaxConcurrentTransfers
    • Transactional

Transfer Optimization 1 - Buffer Size

The BufferSize parameter element defines the maximum data block size. TODO

Transfer Optimization 2 - Concurrent Transfer

The ConcurrentTransfer element is used to allow parallel file transfer to occur.

A child element of ConcurrentTransfer - MaxConcurrentTransfers - can be used to limit the number of parallel transfers.

  • TransferOptions
    • ConcurrentTransfer
      • MaxConcurrentTransfers
       

Transactional Transfer

This parameter specifies whether a transfer should be processed as a single transaction, i.e. whether all objects are to be successfully transferred or none at all.

Should an error occur during a transfer operation then transfers already carried out will be rolled back.

Transactional transfer is often used together with the following parameters:

  • Atomicity (a TargetFileOption for the CopyTarget and MoveTarget elements)
  • DisableOverwriteFiles (a TargetFileOption for the CopyTarget and MoveTarget elements)

See the Transactional Parameter Reference Article for a detailed explanation of the dependencies between these elements.

Note that transactional transfer is not 100% reliable - there are situations where rollback will not be possible.

File Renaming

File renaming is protocol-specific and is therefore specified as part of the in the FragmentRef branch of the profile.

File renaming is specified using the Rename element, which is can be specified as a child element of FragmentRef elements such as FTPFragkentRef, SMBFragmentRef and LocalSource and LocalTarget elements in Profiles.

The Rename element requires that two child elements are defined:

  • ReplaceWhat - a regular expression that allows selection of parts of file names.
  • ReplaceWith - complex replacement patterns using substitution masks and capturing groups can be used.

A detailed description of the possible file name replacement patterns can be found in the Rename element pattern reference.

See also Atomicity.

 

 

  • No labels