JADE Parameter Reference - Operation Copy

• Schema: CopySource, CopySource

This parameter specifies the source of a Copy operation.

The following frequently used settings groups are available:

• CopySourceFragmentRef: references one of the ProtocolFragments, e.g. for transfer by FTP, FTPS, SFTP etc..
• SourceFileOptions: specifies e.g. the Selection of files for transfer.

• Schema: CopySourceFragmentRef, CopySourceFragmentRef

This element references a readable fragment that is used to copy files in a Copy operation.

Any number of reusable Fragments elements can be configured and are distinguished by YADE based on their name attribute.

A subsequent fragment reference element points to the fragment that is used as source in the Copy operation.

• Schema: FTPFragmentRef, FTPFragmentRef

This element references the fragment that is used for an FTP file transfer operation.

Any number of reusable FTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPFragment element.

• Schema: FTPSFragmentRef, FTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable FTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPSFragment element.

• Schema: HTTPFragmentRef, HTTPFragmentRef

This element references the fragment that is used for an HTTP file transfer operation. Any number of reusable HTTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPFragment element.

• Schema: HTTPSFragmentRef, HTTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable HTTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPSFragment element.

• Schema: LocalSource, LocalSource
• Mapping:
  • protocol

A local source is considered to be located locally to the YADE Client, i.e. in the local file system.

• Schema: SFTPFragmentRef, SFTPFragmentRef

This element references the fragment that is used for an SFTP file transfer operation. Any number of reusable SFTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SFTPFragment element.

• Schema: SMBFragmentRef, SMBFragmentRef

This element references the fragment that is used for an SMB file transfer operation. Any number of reusable SMBFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SMBFragment element.

• Schema: WebDAVFragmentRef, WebDAVFragmentRef

This element references the fragment that is used for an WebDAV file transfer operation. Any number of reusable WebDAVFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the WebDAVFragment element.

• Schema: AlternativeCopySourceFragmentRef, AlternativeCopySourceFragmentRef

Alternative fragements can be specified as a replacement for existing file transfer fragments in case that a connection cannot be established with those fragments.

Alternative fragments can be specified from a range of protocol fragments for use with the source of a file transfer operation without modifying files.

For each fragment type depending on the file transfer protocol in use an alternative fragment type is available.

• Schema: FTPFragmentRef, FTPFragmentRef

This element references the fragment that is used for an FTP file transfer operation.

Any number of reusable FTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPFragment element.

• Schema: FTPSFragmentRef, FTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable FTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPSFragment element.

• Schema: HTTPFragmentRef, HTTPFragmentRef

This element references the fragment that is used for an HTTP file transfer operation. Any number of reusable HTTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPFragment element.

• Schema: HTTPSFragmentRef, HTTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable HTTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPSFragment element.

• Schema: LocalSource, LocalSource
• Mapping:
  • protocol

A local source is considered to be located locally to the YADE Client, i.e. in the local file system.

• Schema: SFTPFragmentRef, SFTPFragmentRef

This element references the fragment that is used for an SFTP file transfer operation. Any number of reusable SFTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SFTPFragment element.

• Schema: SMBFragmentRef, SMBFragmentRef

This element references the fragment that is used for an SMB file transfer operation. Any number of reusable SMBFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SMBFragment element.

• Schema: WebDAVFragmentRef, WebDAVFragmentRef

This element references the fragment that is used for an WebDAV file transfer operation. Any number of reusable WebDAVFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the WebDAVFragment element.

• Schema: Directory, Directory
• Mapping:
  • dir

This parameter is used to specify a directory on one of the involved servers. Directories can be adressed using a folder structure, e.g. /folder1/folder2.

For Windows systems when used with the LocalSource parameter then a Windows path including a drive letter can be specified. For all server systems the standard syntax using can be used which includes forward slashes to separate folder names.

The directory can be set absolute or relative to the working directory, where the working directory depends on the server configuration. If for example a user "test" connects to an SFTP server then this user might have /home/test/ as a working directory. This applies to servers that use the home directory as the working directory. In this case you can use relative and absolute adressing as in

• data
• /home/test/data

• Schema: SourceFileOptions, SourceFileOptions

These options apply to the handling of files on a source server. They specify e.g. the Selection of files for Copy and Move operations.

• Schema: Selection, Selection

A file Selection includes specifying one of

• FilePathSelection: an absolute path to a file or a relative path used with a Directory
• FileSpecSelection: a regular expression specifying a pattern for file names that are looked up in a Directory
• FileListSelection: a list of files from a Directory

• Schema: CheckSteadyState, CheckSteadyState
• Mapping:
  • check_steady_state_of_files

In some file transfer scenarios the recipient of a file does not know when the sender creates the file. In case of (very) large files the recipient may try to read the file before the sender has finished writing it. This can result in the recipient retrieving an incomplete file.

Including the CheckSteadyState parameter ensures that the file is checked at its reception point for completeness before starting the transfer.

Including the CheckSteadyState parameter ensures that the file is checked at its reception point for completeness before starting the transfer.

If the check of the steady state of one incoming file did not succeed successfully then no file will be transferred.

Note that this is not a very reliable approach as the recipient checks the date of last modification and the size of the file. If neither changes during a specified a period of time then the file is assumed to be complete. However, if transmission is terminated without the file being completely written, the network goes down or the processing speed of the file is slow, then the receiver will get a corrupted file.

The shorter the check interval is chosen the more likely it may happen that a file is detected as steady, although this was not yet finished writing.

• Schema: Directives, Directives

Directives are options that specify the behavior with specific file characteristics:

• DisableErrorOnNoFilesFound: do not raise an error if no files are found on the source server
• TransferZeroByteFiles: specify whether zero byte files should be transferred and processed by subsequent commands

• Schema: Polling, Polling

Polling is the YADE capability to wait for incoming files.

• If a file is found then polling is completed and the file transfer starts.
• If no file is found then YADE would wait for a configurable PollInterval and then repeat the check for incoming files.
• Having exceeded the PollTimeout YADE will not repeat checks for incoming files but will fail with the respective exception.

• Schema: ResultSet, ResultSet
• Mapping:
  • create_result_set
  • CreateResultSet
  • CreateResultList
  • create_result_list

This handles the result set of a Selection of files from a source host. The result set can be written into a file or checked against an expectation.

• Schema: MaxFiles
• Mapping:
  • Max_Files
  • MaxFiles

This parameter specifies the maximum number of files that are considered for transfer. If additional files were available then they will be ignored.

Note that it is not possible to specify which files will be ignored should the value set for this parameter be exceeded. It therefore only makes sense to use this parameter in particular situations such as when the contents of a directory will be repeatedly polled.

• Schema: CheckIntegrityHash, CheckIntegrityHash
• Mapping:
  • check_security_hash

This parameter makes use of a checksum file that is available with the source of a file transfer. The name of the checksum file is assumed to be the same as the source file with an additional extension .md5. During file transfer the contents of the checksum file is compared with the checksum that is calculated from the transfer of the file.

When used with a jump host then integrity checking applies to source and jump host, not to the target of the transfer.

With this element being used a checksum file is expected on the source system and the integrity hash for the target file is calculated and compared with the respective integrity hash of the source file. If the hashes are not equal then the file transfer will be rolled back. If the checksum file on the source system is missing then this will be reported as an information but will not affect the transfer of files.

• Schema: CopyTarget, CopyTarget

This parameter specifies the target of a Copy operation.

The following frequently used settings groups are available:

• CopyTargetFragmentRef: references one of the ProtocolFragments, e.g. for transfer by FTP, FTPS, SFTP, to a local target etc..
• TargetFileOptions: specifies e.g. if existing files on the target system should be overwritten.

• Schema: CopyTargetFragmentRef, CopyTargetFragmentRef

This element references a writeable fragment that is used to copy files in a Copy operation.

Any number of reusable Fragments elements can be configured and are distinguished by YADE based on their name attribute.

A subsequent fragment reference element points to the fragment that is used as target in the Copy operation.

• Schema: FTPFragmentRef, FTPFragmentRef

This element references the fragment that is used for an FTP file transfer operation.

Any number of reusable FTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPFragment element.

• Schema: FTPSFragmentRef, FTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable FTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPSFragment element.

• Schema: HTTPFragmentRef, HTTPFragmentRef

This element references the fragment that is used for an HTTP file transfer operation. Any number of reusable HTTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPFragment element.

• Schema: HTTPSFragmentRef, HTTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable HTTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPSFragment element.

• Schema: LocalTarget, LocalTarget
• Mapping:
  • protocol

A local target is considered to be located on the server where the YADE Client is operated, i.e. in the local file system.

• Schema: SFTPFragmentRef, SFTPFragmentRef

This element references the fragment that is used for an SFTP file transfer operation. Any number of reusable SFTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SFTPFragment element.

• Schema: SMBFragmentRef, SMBFragmentRef

This element references the fragment that is used for an SMB file transfer operation. Any number of reusable SMBFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SMBFragment element.

• Schema: WebDAVFragmentRef, WebDAVFragmentRef

This element references the fragment that is used for an WebDAV file transfer operation. Any number of reusable WebDAVFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the WebDAVFragment element.

• Schema: AlternativeCopyTargetFragmentRef, AlternativeCopyTargetFragmentRef

Alternative fragements can be specified as a replacement for existing file transfer fragments in case that a connection cannot be established with those fragments.

Alternative fragments can be specified from a range of protocol fragments for use with the source or target of a file transfer operation that required modifying files.

For each fragment type depending on the file transfer protocol in use an alternative fragment type is available.

• Schema: FTPFragmentRef, FTPFragmentRef

This element references the fragment that is used for an FTP file transfer operation.

Any number of reusable FTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPFragment element.

• Schema: FTPSFragmentRef, FTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable FTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the FTPSFragment element.

• Schema: HTTPFragmentRef, HTTPFragmentRef

This element references the fragment that is used for an HTTP file transfer operation. Any number of reusable HTTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPFragment element.

• Schema: HTTPSFragmentRef, HTTPSFragmentRef

This element references the fragment that is used for an FTPS file transfer operation. Any number of reusable HTTPSFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the HTTPSFragment element.

• Schema: LocalTarget, LocalTarget
• Mapping:
  • protocol

A local target is considered to be located on the server where the YADE Client is operated, i.e. in the local file system.

• Schema: SFTPFragmentRef, SFTPFragmentRef

This element references the fragment that is used for an SFTP file transfer operation. Any number of reusable SFTPFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SFTPFragment element.

• Schema: SMBFragmentRef, SMBFragmentRef

This element references the fragment that is used for an SMB file transfer operation. Any number of reusable SMBFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the SMBFragment element.

• Schema: WebDAVFragmentRef, WebDAVFragmentRef

This element references the fragment that is used for an WebDAV file transfer operation. Any number of reusable WebDAVFragment elements can be configured and are distinguished by YADE based on their name attribute.

The reference to a fragment is based on the value of the ref attribute of this element to the corresponding name attribute of the WebDAVFragment element.

• Schema: Directory, Directory
• Mapping:
  • dir

This parameter is used to specify a directory on one of the involved servers. Directories can be adressed using a folder structure, e.g. /folder1/folder2.

For Windows systems when used with the LocalSource parameter then a Windows path including a drive letter can be specified. For all server systems the standard syntax using can be used which includes forward slashes to separate folder names.

The directory can be set absolute or relative to the working directory, where the working directory depends on the server configuration. If for example a user "test" connects to an SFTP server then this user might have /home/test/ as a working directory. This applies to servers that use the home directory as the working directory. In this case you can use relative and absolute adressing as in

• data
• /home/test/data

• Schema: Directory, Directory
• Mapping:
  • dir

This parameter is used to specify a directory on one of the involved servers. Directories can be adressed using a folder structure, e.g. /folder1/folder2.

For Windows systems when used with the LocalSource parameter then a Windows path including a drive letter can be specified. For all server systems the standard syntax using can be used which includes forward slashes to separate folder names.

The directory can be set absolute or relative to the working directory, where the working directory depends on the server configuration. If for example a user "test" connects to an SFTP server then this user might have /home/test/ as a working directory. This applies to servers that use the home directory as the working directory. In this case you can use relative and absolute adressing as in

• data
• /home/test/data

• Schema: TargetFileOptions, TargetFileOptions

These options apply to files on a target server. They specify e.g. the Atomicity of a file transfer for Copy and Move operations.

• Schema: AppendFiles
• Mapping:
  • append_files

This parameter specifies wether the content of a source file should be appended to the target file, should the target file exist.

• By default no file content is appended to the target file wich corresponds to the value "false".
• With a value "true" the content of source files will be appended to the corresponding target files.

If AppendFiles is specified with the value "true" then the parameter DisableOverwriteFiles will be ignored.

• Schema: Atomicity, Atomicity

YADE uses a temporary (atomic) prefix and/or suffix to mask the presence of a file being transferred until the transfer has completed.

• Schema: CheckSize
• Mapping:
  • check_size
  • CheckFileSizeAfterTransfer

This parameter specifies that after transfer of each file to a target server YADE will try to read the file from the target server and compare the number of bytes of the original file on the source server and the file on the target server.

• The default value of this parameter is "true".
• If set to "false" the the size of the source and target file will not be checked.

• Schema: CumulateFiles, CumulateFiles
• Mapping:
  • cumulate_files
  • CumulateFiles

This option determines whether the individual files from the results list should be transferred individually and then should be cumulated (i.e. concatenated into a single file) to the transfer target. The name of the cumulative file is specified using the CumulativeFilename parameter.

The CumulativeFileSeparator parameter is used to specify a string that is added to the target file, between the individual files in order to allow these files to be separated later on.

• By default no files are cumulated which corresponds to the value "false".
• When set to "true" then files will be cumulated.

Should the source files be deleted after transfer then the Move operation can be used.

• Schema: CompressFiles, CompressFiles
• Mapping:
  • compress_files

This parameter specifies whether or not the content of the source files should be compressed for the target files using a zip algorithm.

In case of sending files each file will be compressed in a single zip file. The extension of the filename is configured with the parameter CompressedFileExtension. A gzip-compatible compression is used, no further software components are required.

• By default no files will be compressed which corresponds to the value "false".
• Setting this parameter to the value "true" will cause target files to be compressed.

The following parameters are ignored should CompressFiles be used:

• AppendFiles
• CheckSize

• Schema: CreateIntegrityHashFile
• Mapping:
  • create_security_hash_file
  • CreateSecurityHashFile

This parameter causes a checksum file to be created on the target server that contains the integrity hash value. The file name is created by using the target file name and adding the hash algorithm as an extension, e.g. a target file "myFiles.tar.gz" will cause the integrity hash file "myFiles.tar.gz.md5" to be created when using MD5 with the HashAlgorithm parameter.

• Schema: KeepModificationDate
• Mapping:
  • keep_modification_date

This parameter specifies that YADE will try to assign the modification date of the source file to the target file.

• By default the current date is used when creating or modifying files on a target server which corresponds to the value "false".
• If set to the value "true" then the modification date of each source file is assigned to the target file.

This functionality makes use of commands that are specific for the file transfer protocol, e.g. FTP, SFTP, and is therefore not available for all protocols. In addition server side settings might restrict the use of such commands.

• Schema: DisableMakeDirectories
• Mapping:
  • make_dirs
  • createFoldersOnTarget
  • create_folders_on_target

This parameter specifies that YADE will try to create the directory on the target server that is specified with the Directory parameter should this directory not exist. This parameter has no effect on the subfolders during a recursive transfer. Subfolders of recursive transfer are always created if they do not exist.

• By default target server directories will be created which corresponds to the value "false".
• A value "true" would prevent directories from being created on target servers.

This functionality makes use of commands that are specific for the file transfer protocol, e.g. FTP, SFTP, and is therefore not available for all protocols. Please consider that server side settings might restrict the use of such commands.

• Schema: DisableOverwriteFiles
• Mapping:
  • overwrite_files

This parameter specifies whether existing target files can be overwritten in a file transfer.

• By default target files will be overwritten which corresponds to the value "false".
• A value "true" would prevent target files from being overwritten.

• Schema: TransferOptions, TransferOptions

Transfer options specify the optional behavior of file transfer, e.g. the transactional behavior.

• Schema: BufferSize
• Mapping:
  • buffer_size
  • BufferSize

The maximum size of a data block is defined with this option.

• Schema: Transactional, Transactional
• Mapping:
  • transactional
  • TransactionMode

This parameter specifies whether a transfer should be processed within a single transaction, i.e. either all objects are successfully transferred or none.

Should an error occur during a transfer operation then all transfers will be rolled back.

When specifying the value true then the following applies:

• The AtomicSuffix parameter and AtomicPrefix parameter are used that cause target files to be created with a suffix such as "~" (default) and that causes the respective files to be renamed to their target file name after the transfer of all files has been successfully completed.
• If at least one file out of a set of files cannot be transferred successfully then no files will be renamed, instead the temporarily created files are removed from the target system.
• The parameter remove_files that causes files to be removed after successful transfer will be effective only after all files have been successfully transferred. Otherwise no files will be removed.

• Schema: RetryOnConnectionError, RetryOnConnectionError
• Release:
  • YADE-556 Availability starting with 1.12.13, 1.13.4.

To control reconnect behavior, the YADE has two options:

• RetryCountMax - number of reconnection attempts (for each single file transfer) if there is a connection failure
• RetryInterval - the wait interval between attempts

• Schema: RetryCountMax
• Mapping:
  • connection_error_retry_count_max
• Release:
  • YADE-556 Availability starting with 1.12.13, 1.13.4.

Sets for each single file transfer the number of reconnection attempts if there is a connection failure.

• Schema: RetryInterval
• Mapping:
  • connection_error_retry_interval
• Release:
  • YADE-556 Availability starting with 1.12.13, 1.13.4.

This option specifies the wait interval between attempts.

Possible values:

• <n> - duration in seconds, e.g.: 3s
• <n>m - duration in minutes, e.g.: 2m
• <n>h - duration in hours, e.g.: 1h

Combined values (with blank as separator):

• e.g.: 1m 30s