Introduction

This is the second in a series of articles and describes how to perform simple file selection with the YADE Client via its Command Line Interface.

YADE Tutorials List

YADE Client Command Line Interface

  1. Getting Started and Downloading files
  2. Simple File Selection
  3. More Advanced File Selection
  4. File Transfer

  5. Checking files for completeness
  6. Public / Private Key Authentication
  7. Transfer via a Jump Host / DMZ

YADE Background Service

  1. Installation, Configuration and Use

Prerequisites

See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.

Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.

Download file

The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor from the following link:

Scope of this tutorial

Before starting to transfer files YADE reads the contents of the folder specified using one or more child elements of the Selection parameter.

In the examples in first tutorial in this series a default setting, that all the files in the source directory will be downloaded, was used: .* .

The basic file selection methods will be described in this tutorial:

File Name Filtering and Recursive File Selection

The following profile shows how YADE can be used to:

Configuration structure

ProtocolFragment

The ftp_demo_sos-berlin ProtocolFragment used in the Client Command Line Interface - Getting Started tutorial will be reused.

Profile

The example presented in this tutorial uses a Profile based on the ftp_server_to_local Profile described in the Getting Started tutorial.

The new Profile, has been given the profile_id = ftp_server_to_local_select_recursive and the following changes have been made:

XML Editor Configuration

The Profile Code

The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local_select_recursive">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^test_.\.txt$]]></FileSpec>
                  <Directory><![CDATA[./rec]]></Directory>
                  <Recursive>true</Recursive>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>
[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_select_recursive]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = ^test_.\.txt$
source_dir                          = ./rec
recursive                           = true

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_select_recursive"
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_select_recursive"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_select_recursive"
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_select_recursive"

Behavior

The relevant file structure on the source server is:

Behavior with DisableErrorOnNoFilesFound = true

The default behavior of YADE is to throw an error if a file transfer is attempted and no files are found. The DisableErrorOnNoFilesFound element is used to allow YADE to attempt a file transfer without an error being generated if no files are found. This behavior can be useful when, for example, polling for files.

The DisableErrorOnNoFilesFound element is specified as a Directives child element, the next screenshot shows how this element is integrated in a configuration:

The effect of setting the DisableErrorOnNoFilesFound element can be demonstrated by, for example, setting the FileSpec regular expression in the Profile element to ^text\.txt$ and recalling the operation.

The file transfer will now end with an Exit Code 99.

If, however, the DisableErrorOnNoFilesFound element is set to true, YADE will not throw an error.

Note:

Zero-Byte Files

The YADE Client can handle files with zero bytes in a number of ways, depending on the setting used for the TransferZeroByteFiles parameter, which is specified as a child of the Directives element.

Test Example

The default TransferZeroByteFiles setting is true. The screenshot below shows how this element is integrated in a duplicate of the ftp_server_2_local file transfer profile that was described in the first tutorial in this series. This new Profile has profile_id = ftp_server_2_local_zero_byte.

In its original configuration - i.e. with FileSpec set to .*, the ftp_server_2_local profile transfers six files from the SOS test server to the local file system. One of these files - ./test_4.txt - has zero bytes and can be used to demonstrate the use of the TransferZeroByteFiles parameter.

The screenshot also shows the relevant extract from the parameter reference for the TransferZeroByteFiles element with a description of the meaning of the different values the TransferZeroByteFiles element can have.

XML Editor Configuration

The Profile Code

The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local_zero_byte">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[.*]]></FileSpec>
                  <Directory><![CDATA[./]]></Directory>
                </FileSpecSelection>
              </Selection>
              <Directives>
                <DisableErrorOnNoFilesFound>true</DisableErrorOnNoFilesFound>
                <TransferZeroByteFiles><![CDATA[false]]></TransferZeroByteFiles>
              </Directives>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>
[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_zero_byte]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = .*
source_dir                          = ./
force_files                         = false
zero_byte_transfer                  = false

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_zero_byte"
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_zero_byte"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_zero_byte"
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_zero_byte"

Behavior