The YADE Client Command Line Interface - Tutorial 2 - Simple File Selection

Introduction

This is the second in a series of articles and describes how to perform simple file selection with the YADE Client via its Command Line Interface.

YADE Tutorials List

YADE Client Command Line Interface

1. Getting Started and Downloading files
2. Simple File Selection
3. More Advanced File Selection

4. File Transfer

5. Checking files for completeness
6. Public / Private Key Authentication
7. Transfer via a Jump Host / DMZ

YADE Background Service

1. Installation, Configuration and Use

Prerequisites

See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.

Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.

Download file

The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor from the following link:

• sos-berlin_demo_2_local_select.xml

Scope of this tutorial

Before starting to transfer files YADE reads the contents of the folder specified using one or more child elements of the Selection parameter.

In the examples in first tutorial in this series a default setting, that all the files in the source directory will be downloaded, was used: .* .

The basic file selection methods will be described in this tutorial:

• File name filtering and recursive file selection.
• Handling of zero-byte files.

File Name Filtering and Recursive File Selection

The following profile shows how YADE can be used to:

• Filter the names of files to be transferred using a regular expression
• Transfer files from sub-directories recursively

Configuration structure

ProtocolFragment

The ftp_demo_sos-berlin ProtocolFragment used in the Client Command Line Interface - Getting Started tutorial will be reused.

Profile

The example presented in this tutorial uses a Profile based on the ftp_server_to_local Profile described in the Getting Started tutorial.

The new Profile, has been given the profile_id = ftp_server_to_local_select_recursive and the following changes have been made:

• The value of the file_spec defines the regular expression used to filter file names

• A recursive child element has been added to the FileSpecSelection element and its value set to true (the default value is false).
• The source_dir parameter has also been changed to:
  • /rec

XML Editor Configuration

The Profile Code

The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local_select_recursive">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^test_.\.txt$]]></FileSpec>
                  <Directory><![CDATA[./rec]]></Directory>
                  <Recursive>true</Recursive>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>

[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_select_recursive]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = ^test_.\.txt$
source_dir                          = ./rec
recursive                           = true

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_select_recursive"

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_select_recursive"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_select_recursive"

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_select_recursive"

Behavior

The relevant file structure on the source server is:

• /

  • /rec
    test.txt
    test_1.txt

    • /sd
      test_2.txt

• YADE will download the test_1.txt from the /rec folder on the server and ignore the test.txt file whose name does not match with the FileSpec regular expression.
• YADE will create a /sd folder in the target a directory if this folder does not already exist.
• It will the open the /rec/sd sub-folder on the server, create a corresponding folder on the target and download the test_2.txt file to this folder.

Behavior with DisableErrorOnNoFilesFound = true

The default behavior of YADE is to throw an error if a file transfer is attempted and no files are found. The DisableErrorOnNoFilesFound element is used to allow YADE to attempt a file transfer without an error being generated if no files are found. This behavior can be useful when, for example, polling for files.

The DisableErrorOnNoFilesFound element is specified as a Directives child element, the next screenshot shows how this element is integrated in a configuration:

The effect of setting the DisableErrorOnNoFilesFound element can be demonstrated by, for example, setting the FileSpec regular expression in the Profile element to ^text\.txt$ and recalling the operation.

The file transfer will now end with an Exit Code 99.

If, however, the DisableErrorOnNoFilesFound element is set to true, YADE will not throw an error.

Note:

• The meaning of the values true and false for the DisableErrorOnNoFilesFound element are the opposite to the meaning for the element's settings.ini file predecessor, force_files.

Zero-Byte Files

The YADE Client can handle files with zero bytes in a number of ways, depending on the setting used for the TransferZeroByteFiles parameter, which is specified as a child of the Directives element.

Test Example

The default TransferZeroByteFiles setting is true. The screenshot below shows how this element is integrated in a duplicate of the ftp_server_2_local file transfer profile that was described in the first tutorial in this series. This new Profile has profile_id = ftp_server_2_local_zero_byte.

In its original configuration - i.e. with FileSpec set to .*, the ftp_server_2_local profile transfers six files from the SOS test server to the local file system. One of these files - ./test_4.txt - has zero bytes and can be used to demonstrate the use of the TransferZeroByteFiles parameter.

The screenshot also shows the relevant extract from the parameter reference for the TransferZeroByteFiles element with a description of the meaning of the different values the TransferZeroByteFiles element can have.

XML Editor Configuration

The Profile Code

The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local_zero_byte">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[.*]]></FileSpec>
                  <Directory><![CDATA[./]]></Directory>
                </FileSpecSelection>
              </Selection>
              <Directives>
                <DisableErrorOnNoFilesFound>true</DisableErrorOnNoFilesFound>
                <TransferZeroByteFiles><![CDATA[false]]></TransferZeroByteFiles>
              </Directives>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>

[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_zero_byte]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = .*
source_dir                          = ./
force_files                         = false
zero_byte_transfer                  = false

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_zero_byte"

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_zero_byte"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_zero_byte"

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_zero_byte"

Behavior

• With the TransferZeroByteFiles element set to false or relaxed (and with the FileSpec set to ^test_.\.txt$ as listed above) neither will the test_4.txt zero byte file be downloaded nor will an error be raised.
• Setting the TransferZeroByteFiles element to false and rerunning the example will:
  • Cause the the test_4.txt file to be ignored as with the relaxed setting and
  • Cause an error to be raised if no files with more than zero bytes are found - i.e. if no files are to be transferred.
    This can be demonstrated by changing the FileSpec regex to ^test_[4]\.txt$ which will only match for the zero byte file.
• A setting of TransferZeroByteFiles = strict will cause an error to be raised if any zero byte files are matched.