Introduction

This is the first in a series of articles describing how to get started with using the the JADE YADE Client via its Command Line Interface and configuring simple file management tasks.

Other articles in this series are:

• II - Simple File Selection

...

JADE Command Line Interface

...

Read more ...

JADE API

...

Read more ...

JADE JITL Jobs

...

Read more ...

Requirements

The JADE Client should be installed

• You can download the JADE Client from our JADE Download Page or from SourceForge.
• Installation of JADE on Unix systems is covered in the How to install JADE article.

About these examples

• This article describes operation of the JADE Client from the command line, e.g. by using a batch script. The other methods of operating the Client are:
  
  • using the JADE API
  • using the JobScheduler's JADE JITL Jobs 
• The examples described in this article have been written for use on a Windows computer but can easily be adapted for Unix environments. Settings files for Windows and Unix systems are available for download via the link in the next section.

• The examples presented on this page use public FTP and SFTP servers that we have configured for this purpose on our web site.
  • Both servers have the same address:
    • test.sos-berlin.com
  • A user demo can log on with password demo and download demonstration files. They cannot upload or delete files.
• Comments in JADE settings profiles are added using double semi-colons (";;").

Setting up the examples:

• For Windows users:
  
  • Create a jade_demo folder in your user home directory with a and b sub-folders like this:
    • mkdir %USERPROFILE%\jade_demo
mkdir %USERPROFILE%\jade_demo\a
mkdir %USERPROFILE%\jade_demo\b
      
  • Download the jade_settings_windows.ini file containing the example profiles presented in this article to the %USERPROFILE%/jade_demo folder and rename it to jade_settings.ini
• For Unix users:
  • Create a jade_demo folder in your user home directory with a and b sub-folders like this:

    • mkdir ~/jade_demo
      mkdir ~/jade_demo/a
      mkdir ~/jade_demo/b

  • Download the jade_settings_unix.ini file containing the example profiles presented in this article to the ~/jade_demo folder and rename it to jade_settings.ini
• Copy the appropriate jade_settings_*.ini file (Windows or Unix) to the jade_demo sub-folder in your JADE installation directory.

:

• For Windows users the location for the jade.cmd file is:
  • C:\Program Files\sos-berlin.com\jade\client\bin
  • or the location that was specified when installing JADE
• for Unix users the location for the jade.sh file is:
  • ~/sos-berlin.com/jade/client/bin
  • or the location that was specified when installing JADE

Note that comments in the JADE settings profiles are added using double semi-colons (";;").

Running the examples:

• For the examples is assumed that jade.cmd or jade.sh are available in your path or are prefixed by their path name.
• The command line parameterization is identical for use with the Windows and Unix command line
• Run the examples by entering the following in the command line, e.g. for the first of the examples listed below:

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="ftp_server_2_local"

./jade.sh -settings="${HOME}/jade_demo/jade_settings.ini" -profile="ftp_server_2_local"

The following examples are given for Windows environment. Unix users will find the samples in the respective configuration file jade_settings.ini.

Simple JADE file transfer operations - download by FTP and SFTP

Transferring the contents of a folder by FTP

YADE Tutorials List

YADE Client Command Line Interface

1. Getting Started and Downloading files
2. Simple File Selection
3. More Advanced File Selection

4. File Transfer

5. Checking files for completeness
6. Public / Private Key Authentication
7. Transfer via a Jump Host / DMZ

YADE Background Service

1. Installation, Configuration and Use

Prerequisites

See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.

Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.

Download file

The configuration described in this file can be downloaded and then directly opened in the XML Editor from the following link:

• sos-berlin_demo_2_local.xml

Simple YADE file transfer operations - download by FTP and SFTP

Transferring the contents of a folder by FTP

In this example the contents of a folder on a public FTP server that the SOS GmbH has configured for demonstration purposes are copied and saved to the local file system.

The full configuration for this operation is shown in the XML Editor screenshot below.

Image Added

Configuration structure

The Configurations element shown has two child nodes Fragments and Profiles, each containing a different group of configuration elements.

• The Fragments elements cover the protocol-specific parts of the configuration - in the current configuration these are that the protocol that is to be used, and the connection and authentication. 
  
  • Note that a file transfer operation usually requires two Fragments - one for the source part of the transfer and one for the target. However a target fragment is not required here as the the target is the local file system.
  • Any number of Fragments can be predefined within a configuration and specified as required using their name attribute - in the example configuration this is ftp_demo_sos-berlln. This predefinition of Fragments allows their reuse and keeps the configuration well structured.
  • Note that it is not necessary to specify a port here as YADE will automatically use the required value of 21 for FTP operations unless specified otherwise.
• The Profile elements contain the parameters that are specific to the current operation
  • the Operation to be carried out is copy,
  • the ftp_demo_sos-berlin Fragment that is to be used for the source part of the operation,
  • the  files that are to be transferred:
    •  all files defined using the .* FileSpec
  • the directory in which the files to be transferred is looked for:
    • defined using  / file path,
  • the directory that files are to be transferred to: ${USERPROFILE}\jade_demo\a
    (on a Linux system the equivalent location would be ${HOME}/jade_demo/a ).
• The ref attribute in the FTPFragmentRef element is used to specify the Fragment to be used - in this case for the source part of the operation.

When the configuration is saved it will be validated against the XSD schema specified for the configuration (see XML Editor - Operation for more information)

The structure of the configuration is described in more detail in the YADE User Manual - Configuring and Running File Transfers article and its child articles.

Configuration Files

The configuration will be converted by default from the XML Editor's native XML format to the settings.ini format required for YADE up to versions 1.10 when the configuration is saved in the XML Editor.

The following code boxes can be opened to show the simple configuration described above in XML and in settings.ini formats:

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[.*]]></FileSpec>
                  <Directory><![CDATA[./]]></Directory>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>

[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol             

In this example the contents of a folder on a remote host are downloaded from our demonstration FTP server and saved on your local file system.

Example

The following profile from your settings file will be used to run the example on your computer:

[ftp_server_2_local]
;;
 operation               = copyftp
;;
 source_host
host                                = test.sos-berlin.com
 source_protocoluser                                = demo
password                            = demo

[ftp_server_2_local]
 source_portoperation                           = 21copy

 source_userinclude                      = protocol_fragment_ftp@ftp_demo_sos-berlin
 source_passwordfile_spec                           = demo.*
 source_dir                          = ./
;;
 target_hostprotocol                     = localhostlocal
 target_protocoldir          = local
 target_dir              = ${USERPROFILE}\jade_demo\a

Running the configuration

The simple configuration described above is started from the command line or in a batch file on Windows systems using one of the following calls, where sos-berlin_demo_2_local is the name given to the configuration file on saving, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local.ini" -profile="ftp_server_2_local_pass"

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local.xml" -profile="ftp_server_2_local_pass"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used parameters listed above should not require further description and are the minimum that are required to copy files from the demo user's root directory on the remote source to a local target directory. By default JADE will download all the files it finds in the directory.Run the example by entering the following in the command line:

./jade.shjade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini${HOME}/jade_demo/sos-berlin_demo_2_local.ini" -profile="ftp_server_2_local_pass"

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local.xml" -profile="ftp_server_2_local_pass"

 More information can be found in the YADE User Manual - Configuring and Running File Transfers article.

Behavior

JADE YADE will log onto the remote server and download all the files it finds in the source _dir folder Directory folder (six in total) before stopping.

Note that by default JADE YADE will create the target folder and/or path if it does not exist and if JADE YADE has the necessary permissions. It will also overwrite any files it finds in the target directory with the same names as files it is downloading.

The Note also that the transfer mode JADE YADE uses for FTP transfer can be specified using the transfer_mode parameter (not used in the example). This can be either ascii or binary (the default setting).

...

Transferring the contents of a folder by SFTP with password authentication

This example uses a new profile, sftp_server_2_local_pass, which is based on the ftp_server_2_local profile used in the previous example.

As the new profile name suggests, the SFTP protocol is used in this profile and authentication is by password - key pair authentication is described later in this document.

Example

The profile is included in the jade_settings_windows.ini file as follows:

...

...

the SFTP protocol and password authentication to copy the files in a folder on a public SFTP server that has been set up in parallel to the FTP server used in the first example to the local file system.

Configuration

While this operation is similar to that described in the first example, it is necessary to create a new Fragment element in the configuration. In addition a new Profile element will be created, to keep use of the Profiles elements as flexible as possible.

Fragment element

The new SFTPFragment is generated by selecting the ProtocolFragments element and using the Add child context menu item (right hand mouse click) in the same way as the FTPFragment was added in the previous example. The completed SFTPFragment configuration is shown in the firat screenshot below. Please note:

• It is not necessary to specify a port here as YADE will automatically use the required value of 22 for SFTP unless specified otherwise.
• The new fragment will be addressed with the sftp_demo_sos-berlin_passname attribute.
• The Password element is not a direct descendant of the Authentication element as was the case with FTP - this allows the use of an alternative authentication method (key pair authentication, described in the Public / Private Key Authentication tutorial)

XML Editor Configuration

Image Added

Profile element

The Profile element required for this example will be almost identical to the FTP Profile from the previous example, with the exception that an SFTPFragmentRef element is required this time. In this case we suggest that the following procedure is used:

• generate the new Profile by duplicating the ftp_server_2_local element using the XML Editor's Copy and Paste functions, which are available using the XML Editor's context menu,
• delete the FTPFragmentRef element and add a new SFTPFragmentRef element,
• select the SFTPFragment as the value for the ref attribute of the SFTPFragmentRef element.

The completed sftp_server_2_local_pass

...

profile is shown below:

Image Added

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local.ini" -profile="sftp_server_2_local_pass"

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local.xml" -profile="sftp_server_2_local_pass"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local.ini" -profile="sftp_server_2_local_pass"

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local.xml

Then call this profile by entering the following in the Windows command line:

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="sftp_server_2_local_pass"

Behavior

As with the FTP transfer profile, JADE YADE will log onto the remote server and download all the files (FileSpec = .*) in the source_dir Directory folder before stopping. As with the previous example, six files will be downloaded.