The YADE Client Command Line Interface - Tutorial 3 - More Advanced File Selection

Introduction

This is the third in a series of articles describing how to get started with using the the YADE Client via its Command Line Interface and covers:

• Polling a Source Directory

YADE Tutorials List

YADE Client Command Line Interface

1. Getting Started and Downloading files
2. Simple File Selection
3. More Advanced File Selection

4. File Transfer

5. Checking files for completeness
6. Public / Private Key Authentication
7. Transfer via a Jump Host / DMZ

YADE Background Service

1. Installation, Configuration and Use

Prerequisites

See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.

Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.

Download file

The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor using the following link:

• sos-berlin_demo_2_local_poll.xml

Polling a source directory for a minimum number of files

YADE can poll a source directory at regular intervals until it finds a file name match or until a specified length of time has elapsed. If it finds a file name match with one or more files it will then transfer the matching file(s) and then stop polling.

YADE can also delay transfer until a minimum number of file name matches has been found.

YADE will transfer all those files it has found at the end of the polling interval if the minimum number of matches has not been reached.

Configuration structure

ProtocolFragment

The ftp_demo_sos-berlin ProtocolFragment used in the first tutorial in this series will be reused. 

Profile

The example presented in this tutorial uses a Profile based on the ftp_server_to_local Profile described in the first tutorial in this series.

The new Profile, has been given the profile_id = ftp_server_to_local_poll and the following changes have been made:

FileSpec

The value of the FileSpec element was changed to ^test_.\.txt$. Five files will be found in the transfer source directory.

Polling Elements

The Polling parent element is added as a child of the SourceFileOptions element. In the example profile shown in the XML Editor screenshot below the function of the three polling parameters should be self-explanatory but the following should be noted:

• PollInterval - is specified in seconds (only integer values)

• PollTimeout - is specified in minutes (only integer values)

• MinFiles - optional - can only be used together with PollTimeout

XML Editor Configuration

The Profile Code

The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/jade/JADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local_poll_minfiles">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^test_.\.txt$]]></FileSpec>
                  <Directory><![CDATA[./]]></Directory>
                </FileSpecSelection>
              </Selection>
              <Polling>
                <PollInterval>20</PollInterval>
                <PollTimeout>1</PollTimeout>
                <MinFiles>3</MinFiles>
              </Polling>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>

[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_poll_minfiles]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = ^test_.\.txt$
source_dir                          = ./
poll_interval                       = 20
poll_timeout                        = 1
poll_minfiles                       = 3

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_poll.ini" -profile="ftp_server_2_local_poll_minfiles"

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_poll.xml" -profile="ftp_server_2_local_poll_minfiles"

On Unix systems the profile is called using one of the following commands, depending on the JADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_poll.ini" -profile="ftp_server_2_local_poll_minfiles"

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_poll.xml" -profile="ftp_server_2_local_poll_minfiles"

Example behavior with MinFiles less than the number of files found 

• YADE runs the first polling operation immediately and will transfer all the files found - in this case the 5 test_*.txt files - as more than the minimum number of files has been found. Polling will then be stopped.
• If no files were to be found then the YADE engine would throw an error as long as DisableErrorOnNoFilesFound = true has not been set.

Example behavior with MinFiles more than the number of files found

Change the MinFiles parameter in the XML Editor to 6, save the configuration and rerun the example.

• YADE will find the five test_*.txt files when it polls but will not transfer the files as the minimum number of 6 has not been reached.
• YADE will repeat polling until the time set in the PollTimeout parameter (with the example configuration after 1 minute 20 secs) has expired.
• Note that all the files that have been found will be transferred once the timeout has been reached.