Pre- & Post-Processing Options

YADE can carry out both pre- and post-processing at the source and at the target parts of the transfer.

A typical post-processing operation at the source would be to move or rename a file after it has been copied to the target.

Pre- and post-processing can be triggered for:

  • Transfer operations.
  • Individual file transfers, so when three files are to be transferred, processing would be carried out three times.

In addition, post-processing can be triggered for:

  • File Rename operations.

Pre- and post-processing can be carried out using the FTP, SFTP and local file system protocols: they are not possible with HTTP, HTTPS, SMBF or WebDAV.

Pre- and post-processing commands

The commands that can be executed as part of a pre- or post-processing operation are protocol-dependant:

  • FTP commands have to be used with the FTP protocol and
  • Shell commands have to be used with the SFTP protocol.

Pre- and post-processing operations

The pre- and post-processing operations available are:

  • Pre-processing operations:
    • CommandBeforeFile:
      • commands are executed before each individual file is transferred
    • CommandBeforeOperation
      • commands are executed before the transfer operation is started
  • Post-processing operations:
    • CommandAfterFile:
      • commands are executed after each file transfer has been completed
    • CommandAfterOperation
      • commands are executed after the transfer operation has been completed
    • CommandBeforeRename
      • commands are executed before each individual file is renamed

Note that special variables such as $target_dir and $date are available for pre- or post-processing commands. These variables are listed under the Parameter Reference article for the relevant protocol *FragmentRef element. See the list of links provided in the Parameter reference for pre- and post-processing operations section on this page for more information.

Specifying pre- and post-processing operations

The commands that can be executed as part of a pre- or post-processing operation are protocol specific - meaning that they are specified as child elements of a protocol fragment element such as SFTPFragmentRef.

The XML hierarchy used to specify the pre- and post-processing operations around a typical Copy operation using SFTP would be:

  • Profile
    • Operation
      • CopySource
        • CopySourceFragmetRef
          • SFTPFragmentRef
            • SFTPPreProcessing
              • CommandBeforeFile
              • CommandBeforeOperation
            • SFTPPostProcessing
              • CommandBeforeFile
              • CommandBeforeOperation
              • CommandBeforeRename
            • etc. (optional)

Parameter reference for pre- and post-processing operations

Detailed information about pre- and post-processing operations can be found in the relevant parameter reference article:

Variables

Variables can be used in pre- and post-processing commands.

Such variables provide the path and name of files in a transfer.

Refactored Variables

See YADE-448 - Getting issue details... STATUS

  • File separator (all paths) are provided as follows:
    • \ : if the pre- and post-processing commands are executed on a Windows machine.
    • / : if the pre- and post-processing commands are not executed on a Windows machine.
  • All directory paths provide the path without a trailing slash or backslash.

 

Variables:

  • Directory variables:
    • ${TargetDirFullName}
    • ${SourceDirFullName}
  • Target file variables:
    • ${TargetFileFullName}
    • ${TargetFileRelativeName}
    • ${TargetFileBaseName}
    • ${TargetFileParentFullName}
    • ${TargetFileParentBaseName}
  • Target transfer file variables:
    • ${TargetTransferFileFullName}
    • ${TargetTransferFileRelativeName}
    • ${TargetTransferFileBaseName}
    • ${TargetTransferFileParentFullName}
    • ${TargetTransferFileParentBaseName}
  • Source file variables:
    • ${SourceFileFullName}
    • ${SourceFileRelativeName}
    • ${SourceFileBaseName}
    • ${SourceFileParentFullName}
    • ${SourceFileParentBaseName}
  • Source renamed file variables:
    • ${SourceFileRenamedFullName}
    • ${SourceFileRenamedRelativeName}
    • ${SourceFileRenamedBaseName}
    • ${SourceFileRenamedParentFullName}
    • ${SourceFileRenamedParentBaseName}

 

Directory ${SourceDirFullName}${TargetDirFullName}
The directory where files are stored on the source or target systems 
  • C:\yade
  • \home\test\yade
Target File NameTarget Settings ${TargetFileFullName}${TargetFileRelativeName}${TargetFileBaseName}${TargetFileParentFullName} ${TargetFileParentBaseName}
The name of a file on the target host.


  • without replacing
  • \home\test\yade\test.txt
  • \home\test\yade\sub_1\test_1.txt
  • \home\test\yade\sub_2\test_2.txt

 

  • test.txt
  • sub_1\test_1.txt
  • sub_2\test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt
  • \home\test\yade
  • \home\test\yade\sub_1
  • \home\test\yade\sub_2
  • yade
  • sub_1
  • sub_2
  • replacing=test
  • replacement=x\y
  • \home\test\yade\x\y.txt
  • \home\test\yade\sub_1\x\y_1.txt
  • \home\test\yade\sub_2\x\y_2.txt
  • x\y.txt
  • sub_1\x\y_1.txt
  • sub_2\x\y_2.txt
  • y.txt
  • y_1.txt
  • y_2.txt
  • \home\test\yade\x
  • \home\test\yade\sub_1\x
  • \home\test\yade\sub_2\x
  • x
  • x
  • x
Target Transfer File NameTarget Settings ${TargetTransferFileFullName} ${TargetTransferFileRelativeName} ${TargetTransferFileBaseName} ${TargetTransferFileParentFullName} ${TargetTransferFileParentBaseName}

During transfer of a file its name can temporary differ:

files can be prefixed or suffixed to indicate temporary names.

 

Target Transfer File Name is not depend

of the target replacing setting.

  • not transactional
  • without suffix or prefix
  • \home\test\yade\test.txt
  • \home\test\yade\sub_1\test_1.txt
  • \home\test\yade\sub_2\test_2.txt
  • test.txt
  • sub_1\test_1.txt
  • sub_2\test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt
  • \home\test\yade
  • \home\test\yade\sub_1
  • \home\test\yade\sub_2
  • yade
  • sub_1
  • sub_2
  • transactional=true
  • atomic_suffix=~
  • \home\test\yade\test.txt~
  • \home\test\yade\sub_1\test_1.txt~
  • \home\test\yade\sub_2\test_2.txt~
  • test.txt~
  • sub_1\test_1.txt~
  • sub_2\test_2.txt~
  • test.txt~
  • test_1.txt~
  • test_2.txt~
  • \home\test\yade
  • \home\test\yade\sub_1
  • \home\test\yade\sub_2
  • yade
  • sub_1
  • sub_2
 Source File NameSource Settings${SourceFileFullName}${SourceFileRelativeName}${SourceFileBaseName}${SourceFileParentFullName}${SourceFileParentBaseName}

The name of a file on the source host.

Source File Name is not depend of the source replacing setting.

 
  • C:\yade\test.txt
  • C:\yade\sub_1\test_1.txt
  • C:\yade\sub_2\test_2.txt
  • test.txt
  • sub_1\test_1.txt
  • sub_2\test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt
  • C:\yade
  • C:\yade\sub_1
  • C:\yade\sub_2
  • yade
  • sub_1
  • sub_2

Source File Renamed Name

Source Settings${SourceFileRenamedFullName}${SourceFileRenamedRelativeName}${SourceFileRenamedBaseName}${SourceFileRenamedParentFullName}${SourceFileRenamedParentBaseName}
The name of a file on the source host after Rename Operation.
  • without replacing
No values are provided for the variables
  • replacing=test
  • replacement=x\y
  • C:\yade\x\y.txt
  • C:\yade\sub_1\x\y_1.txt
  • C:\yade\sub_2\x\y_2.txt
  • x\y.txt
  • sub_1\x\y_1.txt
  • sub_2\x\y_2.txt
  • y.txt
  • y_1.txt
  • y_2.txt
  • C:\yade\x
  • C:\yade\sub_1\x
  • C:\yade\sub_2\x
  • x
  • x
  • x

 

Deprecated Variables

See YADE-448 - Getting issue details... STATUS .

  • File separator (all paths) are provided as follows
    • \ : if the pre- and post-processing commands are executed on a Windows machine.
    • / : if the pre- and post-processing commands are not executed on a Windows machine.
  • All directory paths provide the path with a trailing slash or backslash.

 

Variables:

  • Directory variables:
    • ${TargetDirName}
    • ${SourceDirName}
  • Target file variables:
    • $TargetFileName
    • ${TargetFileName}
  • Target transfer file variables:
    • $TargetTransferFileName
    • ${TargetTransferFileName}
  • Source file variables:
    • $SourceFileName
    • ${SourceFileName}
    • $SourceTransferFileName
    • ${SourceTransferFileName}
  • Source renamed file variables:
    • $RenamedSourceFileName
    • ${RenamedSourceFileName}

 Directory${SourceDirName}${TargetDirName}
The directory where files are stored on the source or target systems
  • C:\yade\
  • \home\test\yade\
 Target File Name$TargetFileName${TargetFileName}$TargetTransferFileName${TargetTransferFileName}

The name of a file on the target host (TargetFileName) without replacements being applied.

During transfer of a file its name can temporarily differ (TargetTransferFileName): files can be prefixed or suffixed to indicate temporary names.

Depending on the variables used absolute and relative paths are provided for files.


  • \home\test\yade\test.txt
  • \home\test\yade\sub_1\test_1.txt
  • \home\test\yade\sub_2\test_2.txt

 

  • test.txt
  • sub_1\test_1.txt
  • sub_2\test_2.txt

During transfer files can be prefixed or suffixed to indicate temporary names.

  • Settings: transactional = true, atomic_suffix=~

    • \home\test\yade\test.txt~

    • \home\test\yade\sub_1\test_1.txt~

    • \home\test\yade\sub_2\test_2.txt~

  • Settings: transactional = false

    • \home\test\yade\test.txt

    • \home\test\yade\sub_1\test_1.txt

    • \home\test\yade\sub_2\test_2.txt

  • Settings: transactional = true atomic_suffix=~

    • test.txt~

    • sub_1\test_1.txt~

    • sub_2\test_2.txt~

The target file name including replacements applied with the following sample settings:

  • target_replacing=test
  • target_replacement=x\y

Replacements are not applied to to temporary names of files during transfer.

  • \home\test\yade\x\y.txt
  • \home\test\yade\sub_1\x\y_1.txt
  • \home\test\yade\sub_2\x\y_2.txt
  • x\y.txt
  • sub_1\x\y_1.txt
  • sub_2\x\y_2.txt
  • Settings: transactional = true atomic_suffix=~

    • \home\test\yade\test.txt~

    • \home\test\yade\sub_1\test_1.txt~

    • \home\test\yade\sub_2\test_2.txt~

  • Settings: transactional = true atomic_suffix=~

    • test.txt~

    • sub_1\test_1.txt~

    • sub_2\test_2.txt~

 Source File Name$SourceFileName${SourceFileName}$SourceTransferFileName${SourceTransferFileName}

The name of a file on the source host without replacements being applied.

During transfer of a file the path is tripped from the file name.

  • C:\yade\test.txt
  • C:\yade\sub_1\test_1.txt
  • C:\yade\sub_2\test_2.txt
  • C:\yade\test.txt
  • C:\yade\sub_1\test_1.txt
  • C:\yade\sub_2\test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt

No replacements are applied to source and target file name variables.

  • C:\yade\test.txt
  • C:\yade\sub_1\test_1.txt
  • C:\yade\sub_2\test_2.txt
  • C:\yade\test.txt
  • C:\yade\sub_1\test_1.txt
  • C:\yade\sub_2\test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt
  • test.txt
  • test_1.txt
  • test_2.txt
Source File Name
after Rename Operation
$RenamedSourceFileName${RenamedSourceFileName}Note
If no source replacement is specified then no values are provided fore the variables.   

The source file name with replacements being applied that include the following sample settings:

  • source_replacing=test
  • source_replacement=x\y
  • C:\yade\x\y.txt
  • C:\yade\sub_1\x\y_1.txt
  • C:\yade\sub_2\x\y_2.txt

  • C:\yade\x\y.txt
  • C:\yade\sub_1\x\y_1.txt
  • C:\yade\sub_2\x\y_2.txt

 

 

 

  • No labels