Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 27 Next »

 

Introduction

The selection & handling of the objects being transferred can be divided into three areas: selection, polling and directives.

Checking files for completeness is handled in the next page.

TODO

Vorschlag:

  • Wg. die Anzahl von Themen könnte man diese Seite eher als Übersichtsseite schreiben mit drei Unterseiten:
    • File Selection
    • File Handling
    • File Transfer

Wie weit würden diese Aufteilung für den Anwender Sinn machen?

(Bis jetzt haben wir über source- and target-seitige 'Operationen' gesprochen.)

File Selection

A tutorial covering the use of the dir, file_spec and recursive parameters for the selection of file objects is available.

TODO

This section covers the following questions:

  1. Select an individual file (file_path)
  2. Select files from a folder (file_path)
    1. Select all files from a folder
    2. Select files recursively
  3. Select a dynamic number of files (file_spec)
    1. based on regular expression
  4. Select files by a list (file_list)
    1. Create a list of file names for selection
  5. Restrict a file selection
    1. file size
    2. number of files
    3. skip files from begin of selection
    4. skip file from end of selection
  6. CheckSteadyState (Od. gehört's besser anderswo?)

Machte es Sinn für den User, diesen Themenbereich gleichwertig wie source-seitig zu machen?

File Handling

How to rename files

Files are renamed using the Rename element, which is specified as a child element of FragmentRef elements such as FTPFragkentRef, SMBFragmentRef and LocalSource and LocalTarget elements in Profiles.

The Rename element requires that two child elements are defined:

  • ReplaceWhat - a regular expression that allows selection of parts of file names.
  • ReplaceWith - complex replacement patterns using substitution masks and capturing groups can be used.

A detailed description of the possible file name replacement patterns can be found in the Rename element pattern reference.

See also Atomicity.

How to append one file to another

The AppendFiles element can be used to specify that files are appended to other files (i.e. added to the end of a file already existing on the target directory)

The AppendFiles element is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically below:

  • Profile
    • Operation
      • Copy
        • CopySource
          • Child elements ...
        • CopyTarget
          • Child elements ...
        • TransferOptions
      • Move
        • MoveSource
          • Child elements ...
        • MoveTarget
          • Child elements ...
        • TransferOptions

Files are appended to files with the same name. VERIFY!

Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles will be ignored.

How to compress files

JADE can compress individual files before writing them to the target directory. VERIFY!

The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.

The CompressFiles element is a child element of the TargetFileOptions element, which itself can be specified for CopyTarget and MoveTarget elements in a Profile as shown schematically below.

  • Profile
    • Operation
      • Copy
        • CopyTarget
          • TargetFileOptions
            • CompressFiles
        • etc ...
      • Move
        • MoveTarget
          • TargetFileOptions
            • CompressFiles
        • etc ...

JADE is not yet able to combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).

Could this be done using SFTP and preprocessing?VERIFY!

 

How to handle Zero Byte Files EVTL. MOVE TO SELECTION

The handling of zero byte files is specified with the TransferZeroByteFiles element.

This element specifies whether zero byte files are to be transferred and processed by subsequent commands.

The options available and the interaction with the Remove and DisableErrorOnNoFilesFound elements are described in detail in the TransferZeroByteFiles parameter reference article.

The TransferZeroByteFiles element is a child of the Directives element, which itself is a child of the SourceFileOptions elements.

These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.

How to configure error handling

The default configuration for JADE is that an error will be logged if a transfer should take place but no files are found

This behavior can altered with the DisableErrorOnNoFilesFound element, which is a child of the Directives element, which itself is a child of the SourceFileOptions elements.

These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.

See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.

File Transfer Options

Move Transactional?

Move to Error Handling section?

See also ErrorHandling (above)

File transfer options are used to:

  • optimize file transfer (buffer size, concurrent transfer) and
  • handle errors (transactional transfer)

Specification of File Transfer Options

File Transfer Options are defined using the FileTransferOptions elements. FileTransferOptions elements are specified as part of a file transfer Profile, as child elements of the Copy or Move elements as shown in the following diagram:

  • Profile
    • Operation
      • Copy
        • CopySource
          • Child elements ...
        • CopyTarget
          • Child elements ...
        • TransferOptions
      • Move
        • MoveSource
          • Child elements ...
        • MoveTarget
          • Child elements ...
        • TransferOptions

It follows that FileTransferOptions elements are independent of the file transfer fragment specified and that separate Options can be specified for the source and target parts of the transfer.

The TransferOptions elements can have three children as listed below. The use of these child elements is described in the next sections:

  • TransferOptions
    • BufferSize
    • ConcurrentTransfer
      • MaxConcurrentTransfers
    • Transactional

Transfer Optimization 1 - Buffer Size

The BufferSize parameter element defines the maximum data block size. TODO

Transfer Optimization 2 - Concurrent Transfer

The ConcurrentTransfer element is used to allow parallel file transfer to occur.

A child element of ConcurrentTransfer - MaxConcurrentTransfers - can be used to limit the number of parallel transfers.

  • TransferOptions
    • ConcurrentTransfer
      • MaxConcurrentTransfers
       

Transactional Transfer

This parameter specifies whether a transfer should be processed as a single transaction, i.e. whether all objects are to be successfully transferred or none at all.

Should an error occur during a transfer operation then transfers already carried out will be rolled back.

Transactional transfer is often used together with the following parameters:

  • Atomicity (a TargetFileOption for the CopyTarget and MoveTarget elements)
  • DisableOverwriteFiles (a TargetFileOption for the CopyTarget and MoveTarget elements)

See the Transactional Parameter Reference Article for a detailed explanation of the dependencies between these elements.

Note that transactional transfer is not 100% reliable - there are situations where rollback will not be possible.

 

  • No labels