Product Knowledge Base

Space shortcuts

Page tree

Introduction

This is the fifth in a series of articles describing how to get started with using the YADE Client via its Command Line Interface and covers three methods of checking for file completeness.

YADE Tutorials List

YADE Client Command Line Interface

  1. Getting Started and Downloading files
  2. Simple File Selection
  3. More Advanced File Selection

  4. File Transfer

  5. Checking files for completeness
  6. Public / Private Key Authentication
  7. Transfer via a Jump Host / DMZ

YADE Background Service

  1. Installation, Configuration and Use

Prerequisites

See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.

Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.

Download files

The configurations described in this tutorial can be downloaded and then directly opened in the XML Editor. Links for each configuration are available at the start of each section of this tutorial.

Scope of this Tutorial

A number of methods for checking whether file transfer has been completed are presented in this document:

  • Integrity hash checksums such as MD5, which are generated before file transfer starts and can be used to carry out a before / after comparison of file size after transfer has been completed.  
  • Atomic file transfer, which is used to hide files being transferred from target directory monitors until the file transfer has been completed.
    • A prefix and/or suffix is added to the name of the file for the duration of the transfer.
      This prefix/suffix is chosen so that the file being transferred will not be matched by any target host file monitor.
    • Once transfer has been completed, YADE removes the prefix/suffix and the file becomes visible to a file monitor.
  • Regularly checking the file size until it reaches a steady state. Here it is assumed that the transfer has been completed when two checks in series return the same size.

Other methods are:

  • Repeating the file transfer and comparing the sizes of the file transfers. When both transfers have the same size then it is assumed that the transfer is complete.
    See the CheckSize element documentation for more information.
  • Checking file size after transfer with the CheckSize element.

Use of the transactional parameter is generally recommended if more than one file is to be transferred.

File Completeness - Integrity Hash Checksums

Download file

The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:

Example a) - Creating an integrity hash file

YADE can generate an MD5 checksum file for a file. This file which is given the ending .md5 and can then be transferred along with the 'original' file. Note however that YADE only generates the checksum file during a file transfer operation, whilst it is reading the 'original' file from the transfer source. This means that this feature has a limited scope - in practice YADE is most often used to check files for completeness using integrity hash files that have been generated by other sources.

Note that to create and save an integrity hash, YADE requires write permissions for the target directory. This means that this feature cannot be demonstrated by downloading files from our test server, where the demo user only has read permissions. Instead it will be demonstrated through local transfer.

Note

  • This example uses a set of 5 test_*.txt files that have been saved to the user's local file system during the implementation of an earlier tutorial in this series - for example The YADE Client Command Line Interface - Tutorial 1 - Getting Started in which the files were downloaded to the user's \jade_demo\a directory.
  • A ProtocolFragment is not required in the configuration of this example, as local to local transfer is being carried out.

XML Editor Configuration

The Profile Code

The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.

The 'local_2_local_create_md5' Profile in XML Format 
<?xml version="1.0" encoding="utf-8"?>
<Configurations>
  <Fragments>
    <ProtocolFragments />
  </Fragments>
  <Profiles>
    <Profile profile_id="local_2_local_create_md5">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <LocalSource />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^(test)(_)[0-9]\.txt]]></FileSpec>
                  <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\b]]></Directory>
            <TargetFileOptions>
              <CheckIntegrityHash>
                <CreateIntegrityHashFile>true</CreateIntegrityHashFile>
              </CheckIntegrityHash>
            </TargetFileOptions>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>
The 'local_2_local_create_md5' Profile in settings.ini Format 
[local_2_local_create_md5]
operation                           = copy
source_protocol                     = local
file_spec                           = ^(test)(_)[0-9]\.txt
source_dir                          = ${USERPROFILE}\jade_demo\a
target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\b
check_security_hash                 = true
create_security_hash_file           = true

Running the Profile

This profile is called on Windows systems using the following command, depending on the YADE version being used:

Running the file transfer configuration in Windows format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.ini" -profile="local_2_local_create_md5"

On Unix systems the profile is called using one of the following commands, again depending on the YADE version being used:

Running the file transfer configuration in Unix format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.ini" -profile="local_2_local_create_md5"

Behavior

If you have already run some of the examples described in previous tutorials, you will have a series of files in the ${USERPROFILE}\jade_demo\a directory on your computer, sequentially named test_1.txt through to test_5.txt.

YADE will carry out the following steps when executing the local_2_local_create_md5 Profile:

  • Read each of the files in the source directory matching the FileSpec regular expression into memory
  • Generate an integrity hash file for each 'original' files it finds matching the FileSpec.
  • Save the integrity hash files along with the 'original' files in the transfer target directory (${USERPROFILE}\jade_demo\b ).
  • An MD5 file corresponding to a file test_1.txt would carry the name test_1.txt.md5.

 

Command Line Interface output for generated intefrity hash files
main INFO  13:57:35,512   (SOSFileListEntry.java:302) ::createChecksumFile SOSVfs_I_285: Checksum-Datei erzeugt: 'C:\Use
rs\aa\jade_demo\b\test_1.txt.md5'

The next example shows the use of the integrity hash files to verify files that have been transferred.

Example b) - Checking transferred files against transferred integrity hash files

After you have run the Create and transfer an MD5 checksum file example above you will have a series of files in your target ${USERPROFILE}\jade_demo\b directory together with the corresponding hash files.

When executing the profile in this example YADE will verify the 'original' files that were transferred to the b directory using their integrity hash files before writing the 'original' files to a new target directory ${USERPROFILE}\jade_demo\b\checked.
To test whether the checks actually work, we suggest running the check twice:

  • first as described below and then
  • after modifying the content of the 'original' files to create an integrity hash mismatch.

XML Editor Configuration

The local_2_local_check_md5 profile shown in the screenshot below is almost identical to the previous local_2_local_create_md5 profile - the only significant difference is that it does not have the CreateIntegrityHashFile parameter as the files are already there.

The Profile Code

The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.

The 'local_2_local_check_md5' Profile in XML Format 
<?xml version="1.0" encoding="utf-8"?>
<Configurations>
  <Fragments>
    <ProtocolFragments />
  </Fragments>
  <Profiles>
    <Profile profile_id="local_2_local_check_md5">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <LocalSource />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^(test)(_)[0-9]\.txt]]></FileSpec>
                  <Directory><![CDATA[${USERPROFILE}\jade_demo\b]]></Directory>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\b\checked]]></Directory>
            <TargetFileOptions>
              <CheckIntegrityHash />
            </TargetFileOptions>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>
The 'local_2_local_check_md5' Profile in settings.ini Format 
 [local_2_local_create_md5]
operation                           = copy
source_protocol                     = local
file_spec                           = ^(test)(_)[0-9]\.txt
source_dir                          = ${USERPROFILE}\jade_demo\a
target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\b
check_security_hash                 = true
create_security_hash_file           = true

[local_2_local_check_md5]
operation                           = copy
source_protocol                     = local
file_spec                           = ^(test)(_)[0-9]\.txt
source_dir                          = ${USERPROFILE}\jade_demo\b
target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\b\checked
check_security_hash                 = true

Running the Profile

This profile is called on Windows systems using the following command, depending on the YADE version being used:

Running the file transfer configuration in Windows format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.ini" -profile="local_2_local_check_md5"

On Unix systems the profile is called using one of the following commands, again depending on the YADE version being used:

Running the file transfer configuration in Unix format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.ini" -profile="local_2_local_check_md5"

Behavior

First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.

If all of the 'original' files in the source directory b are intact (i.e. if you do not manipulate any of the files) then:

  • a new integrity hash will be calculated for each of the 'original' files.
  • These new hashes will then checked against those saved in the .md5 files.

If all the hashes are OK then:

  • No information about the results of the integrity checks will be output.
    • See Issue
Behavior in the Event of an Integrity Hash Mismatch

If any of the 'original' files in the source directory b has been corrupted (or if you have manipulated one or more of the files as suggested above) then an error will be thrown:

Integrity Hash error message
main ERROR 14:11:13,763   (SOSFileListEntry.java:1027) ::run SOSVfs_E_229: Fehler. Daten³bertragung nicht m÷glich. Grund: com.sos.JSHelper.Exceptions.JobSchedulerException: Integrity Hash violation. File C:\Users\aa\jade_demo\b\test_3.txt.md5, checksum read: 'b4817065c96c3103e11ab8295a8e0c2c', checksum calculated: 'dbeb552a55aad26cabf17641104bf64f'
main ERROR 14:11:13,765   (SOSDataExchangeEngine.java:1275) ::transfer SOSDataExchangeEngine.TRANSFER_ABORTED
main INFO  14:11:13,766   (SOSFileList.java:506) ::Rollback Rollback aborted files.

Note that:

  • Files that have already been checked and fully transferred will only be deleted from the target directory - in this case the checked directory - if transactional transfer has been specified.
  • File transfers that have not been completed / started when the error occurs will be aborted / rolled back.

File Completeness - Atomic File Transfer

ExampleDescription

The principle of atomic file transfer was described at the beginning of this article.

The local_2_local_atomic profile listed below is will carry out atomic file transfer from our test server to a local folder. Three relatively large files (each 200KB) will be downloaded and theses may be sufficiently large for you to see that they are first of all written to the local file system with the ~ suffix while the download is taking place and that the suffix is removed once the download has bee completed.

Download file

The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:

XML Editor Configuration

The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.

The Profile Code

The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.

The 'ftp_server_2_local_atomic' Profile in XML Format 
<?xml version="1.0" encoding="utf-8"?>
<Configurations>
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">
        <BasicConnection>
          <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
        <BasicAuthentication>
          <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id="ftp_server_2_local_atomic">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref="ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^test_large_.\.txt$]]></FileSpec>
                  <Directory><![CDATA[./large]]></Directory>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a\large]]></Directory>
            <TargetFileOptions>
              <Atomicity>
                <AtomicSuffix><![CDATA[~]]></AtomicSuffix>
              </Atomicity>
            </TargetFileOptions>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>
The 'ftp_server_2_local_atomic' Profile in settings.ini Format 
[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_atomic]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = ^test_large_.\.txt$
source_dir                          = ./large

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a\large
atomic_suffix                       = ~

Running the Atomic Transfer Profile

This profile is called on Windows systems using the following command, depending on the YADE version being used:

Running the file transfer configuration in Windows format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_atomic.ini" -profile="ftp_server_2_local_atomic"

On Unix systems the profile is called using one of the following commands, again depending on the YADE version being used:

Running the file transfer configuration in Unix format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.ini" -profile="ftp_server_2_local_atomic"

Behavior - Atomic File Transfer

When atomic file transfer is specified YADE will add the characters specified in the AtomicPrefix and AtomicSuffix parameters to the file name. Once the file transfer has been completed, YADE deletes the prefix and suffix.

Whilst this procedure may happen too quickly to be observed in a graphic file manager, the following INFO entries should be able to be seen in the command line interface:

main INFO  09:09:18,622   (SOSDataExchangeEngine.java:1050) ::setInfo 3 files found for regexp '^test_large_.\.txt$'.
main INFO  09:09:19,388   (SOSVfsLocalFile.java:413) ::rename SOSVfs_I_150: Datei 'C:\Users\aa\jade_demo\a\large\test_la
rge_1.txt~' umbenannt in 'C:\Users\aa\jade_demo\a\large\test_large_1.txt'.

File Completeness - Check Steady State

Example Description

The local_2_local_check_steady_state profile listed below is intended to be used in YADE together with a second file transfer program that allows the file transfer rate to be restricted. This second file transfer program is used to transfer a series of files from our test server at a relatively slow rate to the local jade_demo/a folder used in previous examples. In parallel, YADE is used to check the size of files arriving in the folder jade_demo/a. YADE will transfer files to the target folder jade_demo/b once a steady state for each file has been found.

WinSCP is an example of a windows file transfer program that allows transfer rate restriction and was used to test this example.

Carry out the following steps to run the example:

  • Prepare your command line interface to run YADE with the local_2_local_check_steady_state profile using the command line call listed below but do not yet start the transfer.
  • The second file transfer program should be configured to allow a maximum rate of around 16 kB/s. This transfer rate will mean that the download of each example file (approx. 220kB ) will take around 15 seconds. This will provide sufficient time for YADE to be started with the local_2_local_check_steady_stateProfile below, find the files being transferred and note that its own file transfer operation cannot be started until the download has been completed.
  • Log onto our test FTP server with the second file transfer program using the same host, user & password as listed in other examples on this page. 
  • Select the three test_large_*.txt files, which you will find in the demo user's large folder.
    • Start downloading the selected files to the jade_demo/a folder.
  • Quickly start YADE with the local_2_local_check_steady_state' profile.

Download file

The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:

XML Editor Configuration

Local to local file transfer is used in the YADE part of this example so a ProtocolFragment element is not required. The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.

The Profile Code

The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.

The 'local_2_local_steady_state' Profile in XML Format 
 <?xml version="1.0" encoding="utf-8"?>
<Configurations>
  <Fragments>
    <ProtocolFragments />
  </Fragments>
  <Profiles>
    <Profile profile_id="local_2_local_steady_state">
      <Operation>
        <Copy>
          <CopySource>
            <CopySourceFragmentRef>
              <LocalSource />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                <FileSpecSelection>
                  <FileSpec><![CDATA[^test_large_.\.txt$]]></FileSpec>
                  <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
                </FileSpecSelection>
              </Selection>
              <CheckSteadyState>
                <CheckSteadyStateInterval>1</CheckSteadyStateInterval>
                <CheckSteadyStateCount>30</CheckSteadyStateCount>
              </CheckSteadyState>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\b]]></Directory>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>
The 'local_2_local_steady_state' Profile in settings.ini Format 
[local_2_local_steady_state]
operation                           = copy
source_protocol                     = local
file_spec                           = ^test_large_.\.txt$
source_dir                          = ${USERPROFILE}\jade_demo\a
check_steady_state_of_files         = true
check_steady_state_interval         = 1
steady_state_count                  = 30
target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\b

Running the Check Steady State Profile

This profile is called on Windows systems using the following command, depending on the YADE version being used:

Running the file transfer configuration in Windows format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_steady_state.xml" -profile="local_2_local_steady_state"
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_steady_state.ini" -profile="local_2_local_steady_state"

On Unix systems the profile is called using one of the following commands, again depending on the YADE version being used:

Running the file transfer configuration in Unix format using an XML (YADE 1.11 and later) or settings.ini file (YADE 1.10 and earlier)
./jade.sh -settings="${HOME}/jade_demo/local_2_local_steady_state.xml" -profile="local_2_local_steady_state"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_steady_state.ini" -profile="local_2_local_steady_state"

Behavior - Check Steady State of incoming files

Once YADE starts executing the profile the command line interface will show a series of messages containing the following:

main INFO  17:41:52,547   (SOSDataExchangeEngine.java:1046) ::setInfo 3 files found for regexp '^test_large_.\.txt$'.
main INFO  17:41:52,548   (SOSDataExchangeEngine.java:1046) ::setInfo checking file(s) for steady state
main INFO  17:41:52,549   (SOSDataExchangeEngine.java:132) ::checkSteadyStateOfFiles steady check (1 of 25): waiting 2s.
main INFO  17:41:54,554   (SOSDataExchangeEngine.java:193) ::checkSteadyStateOfFile steady check (1 of 25): Changed. file size: new = 147456 bytes, old = 114688 bytes. 'C:\Users\aa\jade_demo\a\test_large_1.txt'

The two checkSteadyStateOfFiles INFO messages will repeat until a steady state is noted for each file and the transfer of that file is started. This is shown in the listing below:

main INFO  17:42:00,566   (SOSDataExchangeEngine.java:193) ::checkSteadyStateOfFile steady check (4 of 25): Changed. file size: new = 224498 bytes, old = 212992 bytes. 'C:\Users\aa\jade_demo\a\test_large_1.txt'
main INFO  17:42:00,567   (SOSDataExchangeEngine.java:132) ::checkSteadyStateOfFiles steady check (5 of 25): waiting 2s.
main INFO  17:42:02,567   (SOSDataExchangeEngine.java:143) ::checkSteadyStateOfFiles all files seem steady! Extra steady check (5 of 25): waiting 2s for late comers.
main INFO  17:42:04,573   (SOSDataExchangeEngine.java:193) ::checkSteadyStateOfFile steady check (5 of 25): Changed. file size: new = 98304 bytes, old = 224498 bytes. 'C:\Users\aa\jade_demo\a\test_large_2.txt'
main INFO  17:42:04,574   (SOSDataExchangeEngine.java:132) ::checkSteadyStateOfFiles steady check (6 of 25): waiting 2s.

After the file being transferred appears to have reached a steady state YADE waits an extra interval to ensure that the file has reached steady state (in the third line of the listing).

In the fourth line YADE is waiting for the next file. Note that the entry old = 224498 bytes comes because in the configuration used to generate the listing, file is being overwritten and 224498 bytes is the size of the previously downloaded version.

Note that it may be necessary to adjust the values of check_steady_state_interval and/or steady_state_count from the values shown to ensure that the test files are fully downloaded before the check is timed out.