YADE Start-up

YADE is the Managed File Transfer utility that ships from a Command Line Client and from a JITL template job used with JS7 Agents. For use of the template job see JS7 - JITL YADEJob.

Start Script: yade.sh, yade.cmd

The YADE Command Line Client includes a Start Script for execution of the Client. The YADE Client ships

• as a standalone client independently from other JS7 products from a separate download,
• included with the JS7 Agent.

The default location of the Start Script is

• for the standalone YADE Client:

  • /opt/sos-berlin.com/js7/yade/bin/yade.sh on Unix and

  • C:\Program Files\sos-berlin.com\js7\yade\bin\yade.cmd on Windows.
• for the YADE Client included with the JS7 Agent:

  • /opt/sos-berlin.com/js7/agent/yade/bin/yade.sh on Unix and

  • C:\Program Files\sos-berlin.com\js7\agent\yade\bin\yade.cmd on Windows.

Use of the Start Script is authoritative to start the YADE Client.

FEATURE AVAILABILITY STARTING FROM RELEASE 2.8.0

Usage

Running the YADE Start Script without arguments displays the usage clause:

Usage: yade.sh|yade.cmd options [switches]
  Options:
    Transfer Options (required):
      --settings=<location of the settings xml file>
      --profile=<profile id>
    Transfer Options:
      --source-dir=<...>
      --source-excluded-directories=<...>
      --source-file-path=<...>
      --source-file-spec=<...>
      --source-file-list=<...>
      --source-recursive=<true|false>
      --target-dir=<...>
    Processing Options:
      --settings-replacer-case-sensitive=<boolean>  | default: true
      --settings-replacer-keep-unresolved=<boolean> | default: true
      --parallelism=<integer>                       | default: 1
    Switches:
      -h | --help                                   | displays usage
      -source-recursive                             | true

Additionally, the following options and switches are available from the Start Script:

Usage:
  Options:
    Startup Options:
      --java-options=<...>
      --return-values=<path>
      --log-level=<info|warn|error|debug|trace>
    Switches:
      -v | --verbose                                 | displays verbose information about invocation of the YADE Client 

Syntax

• Recommended use of options and switches is stated with the usage: options start with a -- double minus and include a - single minus as a word separator.
• For compatibility with JS1 YADE (branch 1.13) options and switches can be stated with a - single minus and with _ underscore as a word separator, for example:
  • --settings, -settings
  • --return-values, -return_values
  • --java-options, -java-options
• The --source-excluded-directories, --source-file-path, --source-file-spec, --source-file-list and --source-recursive options can be specified without the source prefix and using underscore instead of dash as the word separator, for example:
  • --source-file-list, --file-list, -file-list, -file_list
  • --source-file-spec, --file-spec, -file-spec, -file_spec

Transfer Options

• --settings
  • Specifies the path to an XML file holding the file transfer configuration. Typically the configuration is managed with the JOC Cockpit inventory, see JS7 - File Transfer Configuration. For JS7 - File Transfer Deployment the configurations are mapped to JS7 - Job Resources.
  • When executing the YADE Client from a job, users assign the Job Resource holding the file transfer configuration to the job. The JS7 Agent will make the Job Resource available from a temporary file in its <agent-data>/work/values directory. The path to the file is provided from an environment variable with the default name: YADE_XML.
    • For Unix users can specify --settings=$YADE_XML. This is the default value for operation with jobs.
    • For Windows users can specify --settings=%YADE_XML%. This is the default value for operation with jobs.
  • When executing the YADE Client standalone, users specify the path to the YADE XML configuration file like this: --settings=<path-to-xml>.
  • Use of the option for the standalone Client is required, for use with jobs the above default value applies.
• --profile
  • Specifies the name of the profile holding the file transfer configuration. The profile is a section in the XML file transfer configuration that specifies the source and target of a transfer.
  • Use of the option is required.
• --source-dir
  • Specifies the the source directory of a transfer.
  • The option overwrites an existing source directory setting in the YADE XML configuration and is applied to any source protocol such as local, ftp, sftp etc.
• --source-excluded-directories
  • Specifies a number of source directories that should be excluded from recursive file transfer.
  • The option overwrites an existing excluded source directory setting in the YADE XML configuration and is applied to any source protocol such as local, ftp, sftp etc.
• --source-file-path
  • Specifies the path to an individual file that will be selected for transfer.
  • The option overwrites an existing source file setting in the YADE XML configuration and is applied to any source protocol such as local, ftp, sftp etc.
• --source-file-spec
  • Specifies a regular expression to select a number of files for transfer, for example:
    • The --source-file-spec="\.csv$" option will transfer files with the extension .csv.
      • For Unix Shell the $ character is considered a variable reference if followed by other characters. Consider use of single quotes instead of double quotes when specifying regular expressions, for example: --source-file-spec='\.csv$'
    • The --source-file-spec="^ProductDemo_\d{2,4}" option will transfer files with names starting with ProductDemo_ followed by 2-4 digits.
  • The option overwrites an existing source file specification setting in the YADE XML configuration and is applied to any source protocol such as local, ftp, sftp etc.
• --source-file-list
  • Specifies the path to a file that holds the list of file names or paths that will be selected for transfer.
    • Example: the --source-file-list=/home/sos/transfer.txt option specifies a file with the following content:
      • /var/transfer/file1.csv
      • /home/sos/file2.txt
    • At run-time the files specified in transfer.txt will be transferred.
  • The option overwrites an existing source file list setting in the YADE XML configuration and is applied to any source protocol such as local, ftp, sftp etc.
• --source-recursive
  • Specifies that source directories should be looked up recursively and should be created with the target of transfer. Default: false.
  • When used with the --source-file-path option, then the files specified will be transferred. If the --source-dir option is specified, then any sub-directories indicated with the --source-file-path option will be created with the target of transfer. If the --source-dir option is not used, then any directories specified with the --source-file-path option will be created withe the target of tansfer.
  • When used with the --source-file-spec option, then sub-directories of the directory specified using the --source-dir option will be looked up recursively. The sub-directories will accordingly be created with the target of transfer.
  • As an alternative to the option the following switch can be used: -source-recursive.
• --target-dir
  • Specifies the the target directory of a transfer.
  • The option overwrites an existing target directory setting in the YADE XML configuration and is applied to any target protocol such as local, ftp, sftp etc.

Processing Options

• --settings-replacer-case-sensitive
  • Specifies if case-sensitivitiy should be considered when replacing variables included in a YADE XML configuration:
    
    • The true value (default) performs replacements with case-sensitive consideration of variable names.
    • The false value performs replacements with case-insensitive consideration variable names.
  • Variables in a YADE XML configuration are specified like this: ${variable}.
• --settings-replacer-keep-unresolved
  • Specifies if variables included in a YADE XML configuration that cannot be replaced by workflow variables should remain unresolved for replacement by OS environment variables.
  • When operated from a job, then variables in a YADE XMl configuration are
    • first replaced by workflow variables,
    • then replaced by OS environment variables.
  • The default value of the option is true wich allows replacements from both workflow variables and OS environment variables.
• --parallelism
  • Specifies if transfer of a number of files should be performed in parallel.
  • By default files are transferred sequentially, corresponding to use of --parallelism=1.

Startup Options

Startup options are applied by the YADE Start Script. They are not directly passed to the YADE Client, but are used to populate environment variables considered by the YADE Client, see Environment Variables.

• --java-options
  • Specifies Java options that can be used for example to limit Java heap space consumption using --java-options="-Xmx64m".
  • If more than one Java option is specified, then they must be separated by spaces. The value of the option must be quoted.
  • Default value: -Xmx32m
• --return-values
  • When operated from a job, specifies the path to a temporary file that holds return values. The YADE Client creates a return variable that is used by the JS7 - History Service to populate the JS7 - File Transfer History.
  • By default the option is assigned the value of the JS7_RETURN_VALUES environment variable. The variable is created by JS7 Agents and specifies a temporary file holding key/value pairs from which dynamic variables for subsequent jobs are created.
• --log-level
  • Specifies the log level applied to the YADE Client. Applicable values include one of: info, warn, error, debug, trace.. The value corresponds to the Log4j log level explained from JS7 - Log Levels and Debug Options.
  • Values can be specified using lowercase letters or uppercase letters.

Switches

• -h | --help
  • Displays the usage clause.
• -v | --verbose
  • Displayes verbose output about invocation of the YADE Client from the YADE Start Script.
• -source-recursive
  • Recursively considers source sub-directories, see --source-recursive option.

Environment Variables

Start Script Environment Variables

A number of environment variables are considered by the YADE Start Script.

• Environment variables have to be set prior to execution of the Start Script.
  • For Unix Shell consider to export variables like this: export YADE_TZ=Europe/London
• For some environment variables corresponding command line options exist that will take precedence if specified.
  
  

Agent Environment Variables

In addition to environment variables provided by the YADE Start Script a number of environment variables are made available by the JS7 Agents for jobs:

Examples

The examples below explain a number of typical use cases.

Running the YADE Client standalone

/opt/sos-berlin.com/js7/yade/bin/yade.sh --settings=/home/sos/yade.xml \
                                         --profile=copy-sftp

"C:\Program Files\sos-berlin.com\js7\yade\bin\yade.cmd" --settings=C:\ProgramData\sos-berlin.com\yade\yade.xml ^
                                                        --profile=copy-sftp

Explanations:

• The YADE XML configuration file is specified with the --settings option for the location to which users did store the file. The file can be downloaded from the JOC Cockpit Inventory->File Transfer view.
• Consider use of quoting when specifying paths that include spaces.
• Examples make use of line continuation characters. Alternatively, options and switches can be specified from a single command line.

Running the YADE Client from a Job

$JS7_YADE_BIN/yade.sh --settings=$YADE_XML \
                      --profile=copy-sftp

"%JS7_YADE_BIN%\yade.cmd" --settings=%YADE_XML% ^
                          --profile=copy-sftp

Explanations:

• The JS7_YADE_BIN environment variable is made available by the JS7 Agent and points to the <agent-home>/yade/bin directory. For Windows environments consider quoting if spaces are included with the path to YADE.
• The YADE_XML environment variable is created by the Job Resource that is assigned the job and that holds the XML file transfer configuration.

Running the YADE Client with Start-up Options

/opt/sos-berlin.com/js7/yade/bin/yade.sh --settings=/home/sos/yade.xml \
                                         --profile=copy-sftp \
                                         --java-options="-Xmx256m -Xms64m" \
                                         --log-level=debug

"C:\Program Files\sos-berlin.com\js7\yade\bin\yade.cmd" --settings=C:\ProgramData\sos-berlin.com\yade\yade.xml ^
                                                        --profile=copy-sftp ^
                                                        "--java-options=-Xmx256m -Xms64m" ^
                                                        --log-level=debug

Explanations:

• The -Xmx256m -Xms64m Java options are specified if the transfer operation is assumed to process a larger number of files. Values do not depend on file size but on the number of expected files. A value as high as 256 MB would indicate >10000 files being expected for a single transfer.
  • For Windows consider quoting the option and its value.
• The debug log level is specified to receive more detailed output of the YADE Client. This can be helpful for analysis if the connection to a source or target fails.

Running the YADE Client with Transfer Options

Transfer Options are applied to overwrite settings in the YADE XML configuration file. Transfer options can improve reusability of profiles as the same profile can be used with different settings per execution of the YADE Client.

/opt/sos-berlin.com/js7/yade/bin/yade.sh --settings=/home/sos/yade.xml \
                                         --profile=copy-sftp \
                                         --source-directory=/home/sos1/outgoing \
                                         --target-directory=/home/sos2/incoming

"C:\Program Files\sos-berlin.com\js7\yade\bin\yade.cmd" --settings=C:\ProgramData\sos-berlin.com\yade\yade.xml ^
                                                        --profile=copy-sftp ^
                                                        --source-directory=/home/sos1/outgoing ^
                                                        --target-directory=/home/sos2/incoming

Explanations:

• The --source-directory and --target-directory Transfer Options are applied to overwrite related settings in the copy-sftp profile.
• A single copy-sftp profile can be used for a number of file transfers using different directories.

Running the YADE Client with Variable Replacements

Replacements are applied to dynamically specify settings for the YADE XML configuration file at run-time. Replacements can improve reusability of profiles as the same profile can be used with different settings per execution of the YADE Client.

The below examples assume the following variables being used in the YADE XML configuration:

• <Hostname><![CDATA[${SFTP_HOST}]]></Hostname>
• <FileSpec><![CDATA[${FILE_SPEC}]]></FileSpec>
  
  

export SFTP_HOST=localhost
export FILE_SPEC=".*\.$(TZ=Europe/London date +'%Y-%m-%d')\.csv$"

/opt/sos-berlin.com/js7/yade/bin/yade.sh --settings=/home/sos/yade.xml \
                                         --profile=copy-sftp

$env:SFTP_HOST="localhost"
$env:FILE_SPEC=".*\.$(Get-Date ([System.TimeZoneInfo]::ConvertTimeFromUtc((Get-Date).ToUniversalTime(), (Get-Timezone -Id "GMT Standard Time"))) -Format 'yyyy-MM-dd')\.csv`$"

"C:\Program Files\sos-berlin.com\js7\yade\bin\yade.cmd" --settings=C:\ProgramData\sos-berlin.com\yade\yade.xml ^
                                                        --profile=copy-sftp ^

Explanations:

• The example makes use of individual enviornment variables created by the user that will be replaced in the YADE XML configuration as run-time.
• The SFTP_HOST environment variable is set to a hostname.
  • The copy-sftp profile is assumed to reference a protocol fragment that holds the following value for the hostname: ${SFTP_HOST}.
  • Consider use of case-sensitive spelling unless the --settings-replacer-case-sensitive=false option is used.
• The FILE_SPEC environment variable. specifies a regular expression to select files for transfer from a directory. 
  • The copy-sftp profile is assumed to reference a file specification that holds the following value for the hostname: ${FILE_SPEC}.
  • The environment variable's value evaluates for example to the regular expression: .*\.2025-04-20\.csv$
  • The regular expression is intended to filter files that hold the current date in the London time zone in their file name and the .csv extension. For example, accounting.2025-04-20.csv.