Introduction

The YADE File Transfer capabilities cover a larger number of use cases. At the same time users might consider using alternative tools as an equivalent solution for certain types of file transfers:

• The curl command to download or upload files can be considered a file transfer.
• Use of operating system commands such as cp and mv to copy and to move files can be considered a file transfer.
• Individual scripts for similar purposes can be considered a file transfer.

While file transfers from commands and from scripts can easily be added to shell jobs they lack support for the JS7 - File Transfer History:

• The history keeps track of transfers and holds metadata such as the point in time of transfer and duration of transfer.
• The history holds the details for each transferred file.
• The history offers the functionality to search for transferred files.

The JS7 offers the File Transfer History Script to support 3rd party utilities in populating the File Transfer History.

FEATURE AVAILABILITY STARTING FROM RELEASE 2.5.2

JS-2036 - Getting issue details... STATUS

Users of earlier JS7 releases find a patch for download and instructions how to apply the patch for JS7 Agents from the above link.

File Transfer History Script

The File Transfer History Script is available with JS7 Agents from the JS7_AGENT_HOME/bin directory.

Usage

Invoking the File Transfer History Script without arguments displays the usage clause:

Usage: file_transfer_history.sh|.cmd [Options] [Switches]

  Options:
    --transfer-file=<transfer-spec>    | optional: the transfer specification can occur any number of times and is made up of the elements:
                                                   <source-file>,<target-file>[,<file-size>[,<error-message>]]
    --operation=<operation-spec>       | optional: operation to copy, to move, to remove or to get a list of files:
                                                   copy|move|remove|getlist, default: copy
    --source-account=<account>         | optional: account used for authentication with the system holding source files
    --source-protocol=<protocol-spec>  | optional: one of the protocols for access to source files:
                                                   local|ftp|ftps|sftp|ssh|http|https|webdav|webdavs|smb, default: local
    --source-host=<hostname>           | optional: hostname, IP address or URL of the host holding source files, default: localhost
    --source-port=<number>             | optional: port used to connect to the system holding source files, default: 0
    --target-account=<account>         | optional: account used for authentication with the system holding target files
    --target-protocol=<protocol-spec>  | optional: one of the protocols for access to target files:
                                                   local|ftp|ftps|sftp|ssh|http|https|webdav|webdavs|smb, default: local
    --target-host=<hostname>           | optional: hostname, IP address or URL of the host holding target files, default: localhost
    --target-port=<number>             | optional: port used to connect to the system holding target files, default: 0
    --start-time=<date-time>           | optional: ISO date and time for start time of transfer: yyyy-MM-dd hh:mm:ssZ
    --end-time=<date-time>             | optional: ISO date and time for end time of transfer: yyyy-MM-dd hh:mm:ssZ, default: current time
    --error=<string>                   | optional: error message indicating a failed file transfer
    --delimiter=<character>            | optional: delimiter character for entries in the transfer specification, default: comma

   Switches:
    -h | --help                        | displays usage
    --display-args                     | displays command line arguments
    --display-result                   | displays execution result     
    --dry-run                          | sets --display-args and --display-result and does not create entries for the File Transfer History

Options

• --transfer-file
  • Specifies transfer details per file. This option is specified repeatedly if multiple files are transferred.
  • Details are specified from the following elements separated by comma as a delimiter:
    • <source-file>: the full path to the source file
    • <target-file>: the full path to the target file
    • <file-size>: optionally the file size is specified in bytes
    • <error-message>: optionally an error message is specified signaling a failed transfer
  • The delimiter between elements can be specified using --delimiter.
  • Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
  •  Example:
    • --transfer-file="/source/a.txt,/target/a.txt" --transfer-file="/source/b.txt,/target/b.txt,1024,Permission denied"
    • First occurrence signals successful transfer of the file, the second occurrence adds an error message signaling failure.
• --operation
  • Specifies the operation applied to the file transfer which is one of: copy, move, remove, getlist.
  • By default the copy operation is assumed.
• --source-account
  • Specifies the account that is used when authenticating with the system which holds the source files.
• --source-protocol
  • Specifies the protocol used to access source files which is one of:
    • local:  access files using the local file system
    • ftp: access files using the FTP protocol
    • ftps: access file using the FTPS protocol
    • sftp: access files using the SFTP protocol
    • ssh: access system using the SSH protocol
    • http: access files using the HTTP protocol
    • https: access files using the HTTPS protocol
    • webdav: access files using the WebDAV protocol
    • webdavs: access files using the WebDAVs protocol
    • smb: access files using the SMB (CIFS) protocol
• --source-host
  • Specifies the hostname, IP address or URL of the host holding source files.
  • Examples:
    • localhost:  specifies the hostname
    • 127.0.0.1: specifies the IP address of the host
    • localhost:80: specifies the hostname and port
    • ftp://user@localhost:21: specifies the protocol, account, hostname and port
• --source-port
  • Specifies the port used to connect to the system holding source files.
• --target-account
  • Specifies the account that is used when authenticating with the system which holds the target files.
• --target-protocol
  • Specifies the protocol used to access target files.
  • For possible values see --source-protocol.
• --target-host
  • Specifies the hostname, IP address or URL of the host holding target files.
  • For examples see --source-host.
• --target-port
  • Specifies the port used to connect to the system holding target files.
• --start-time
  • ISO date and time of transfer begin. The patterns yyyy-MM-dd hh:mm:ssZ and yyyy-MM-dd'T'hh:mm:ssZ are accepted.
  • Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
  • Examples:
    • 2023-01-01 12:00:00:  noon on 1st of January 2023, the time zone of the server operating the JS7 Agent will be used.
    • 2023-01-01 12:00:00+0100:  noon on 1st of January 2023 in a time zone UTC+1
    • 2023-01-01T12:00:00+0100: same as the previous example, the 'T' character is used to separate date and time.
• --end-time
  • ISO date and time of transfer end. The patterns yyyy-MM-dd hh:mm:ssZ and yyyy-MM-dd'T'hh:mm:ssZ are accepted.
  • Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
  • By default the current date and time will be used.
  • For examples see --start-time.
• --error
  • Specifies an error message that indicates a failed transfer. The JS7 File Transfer History will qualify the transfer being failed.
  • Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
  • Example
    • --error="Permission denied"
• --delimiter
  • Specifies the delimiter character for entries in the transfer specification
  • Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
  • By default the comma is used as a delimiter.
  • Example
    • --delimiter=";" --transfer-file="/source/a.txt;/target/a.txt,512"

Switches

• -h | --help
  • Displays usage.
• --display-args
  • Displays the command line arguments passed to the File Transfer History command line application.
• --display-result
  • Displays the resulting File Transfer History values based on the the command line arguments.
• --dry-run
  • Sets --display-args and --display-result and does not create entries for the File Transfer History.

Exit Codes

• 0 - Signals success.
• 1 - Signals failed execution.
  • Possible reasons:
    • the environment variable JS7_RETURN_VALUES is not set.
    • the File Transfer History result can't be serialized/written to a temporary file indicated with the JS7_RETURN_VALUES environment variable.

Examples

The following examples illustrate typical use cases.

Simple Transfer

#!/bin/sh

cp -p /tmp/source/a/a1.txt /tmp/target/a/

$JS7_AGENT_HOME/bin/file_transfer_history.sh \
    --transfer-file="/tmp/source/a/a1.txt,/tmp/target/a/a1.txt"

# copy a file and create an entry in the JS7 File Transfer History

Copy single File

#!/bin/sh

cp -p /tmp/source/a/a1.txt /tmp/target/a/

$JS7_AGENT_HOME/bin/file_transfer_history.sh \
    --source-account="$(id -u -n)" \
    --target-account="$(id -u -n)" \
    --target-host="$(hostname -a)" \
    --target-protocol="local" \
    --transfer-file="/tmp/source/a/a1.txt,/tmp/target/a/a1.txt,42"

# copy a file and create an entry in the JS7 File Transfer History
# optional arguments for accounts used with source and target systems are specified 

Copy single File (from variables)

#!/bin/sh

source_dir=${DIR}/source/a
source_file=${source_dir}/a1.txt
target_dir=${DIR}/target/a

cp -p "${source_file}" "${target_dir}"/

$JS7_AGENT_HOME/bin/file_transfer_history.sh \
    --source-account="$(id -u -n)" \
    --target-account="$(id -u -n)" \
    --target-host="$(hostname -a)" \
    --target-protocol="local" \
    --transfer-file="${source_file},${target_dir}/$(basename ${source_file}),$(wc -c "${source_file}" | cut -d' ' -f 1)"

# copy a file and create an entry in the JS7 File Transfer History
# the example corresponds to the previous one but makes use of environment variables

Copy multiple Files

#!/bin/bash

source_files=$(ls /tmp/source/a/*)
target_dir=/tmp/target/a

args=()
args_count=0

for source_file in ${source_files[@]}; do
    cp -p "${source_file}" "${target_dir}"
    args[${args_count}]="--transfer-file=${source_file},${target_dir}/$(basename "${source_file}"),$(wc -c "${source_file}" | cut -d' ' -f 1)"
    args_count=$(($args_count + 1))
done

$JS7_AGENT_HOME/bin/file_transfer_history.sh \
    --source-account="$(id -u -n)" \
    --target-account="$(id -u -n)" \
    --target-host="$(hostname -a)" \
    --target-protocol="local" \
    "${args[@]}"

# copy a number of files and create corresponding entries in the JS7 File Transfer History
# for each file a --transfer-file argument is used

Use with Workflows

Integration of the File Transfer History Script with workflows is offered for a number of use cases.

Configuration View

The following workflow example can be uploaded to the Configuration view. The example includes a number of jobs which perform file transfers using Unix cp commands

Download (upload .json): pdTransferHistory.workflow.json

Explanation:

• The first job create-files performs what the title suggests.
• The second job transfer-cp-single performs transfer of a single file which is straightforward.
• The second job transfer-cp-multi performs transfer of a number of files and introduces more complexity.
  • For consistency of transfer the list of available files is created and is used for copy operations and for populating the File Transfer History.
• The third job transfer-cp-multi-error performs transfer of a number of files with individually failed transfers.
  • The job implements the scenario that the second file out of three files cannot be copied.
  • The example is somewhat more complex and demonstrates the limits of OS utilities when it comes to error handling.
  • The amount of shell code required for proper error handling is increasing and exceeds by far the configuration used for a YADE file transfer.

File Transfer View

File transfers performed by the above workflow are displayed in the File Transfer view:

Explanation:

• The first history entry
  • is in a failed state as one out of three transfers failed,
  • displays individual files with source, target and transfer status.
• The second history entry
  • is in a successful state as all transfers succeeded,
  • displays individual files with source, target and transfer status.
• Search operations as offered from the Advanced Filter button can be applied that offer to search for individual files.