Introduction
This is the fourth in a series of articles describing how to get started with using the JADE Client via its Command Line Interface.
The articles in this series are:
- Getting Started and Downloading files
- Simple File Selection
- More Advanced File Selection
- File Transfer
- Checking Files for Completeness
- Public / Private Key Authentication
For setting-up and running the examples see Using the tutorials with the JADE Client Command Line Interface
Checking files for completeness
A number of methods for checking whether file transfer has been completed are presented in this document:
- Atomic file transfer, which is used to hide files being transferred from target directory monitors until the file transfer has been completed.
- A prefix and/or suffix is added to the name of the file for the duration of the transfer.
This prefix/suffix is chosen so that the file being transferred will not be matched by any target host file monitor. - Once transfer has been completed, JADE removes the prefix/suffix and the file becomes visible to a file monitor.
- A prefix and/or suffix is added to the name of the file for the duration of the transfer.
- MD5 hash checksums, which are generated before file transfer starts can be used to check files after transfer.
- Regularly checking the file size until it reaches a steady state. Here it is assumed that the transfer has been completed when two checks in series return the same size.
Other methods are:
- Repeating the file transfer and comparing the sizes of the file transfers. When both transfers have the same size then it is assumed that the transfer is complete.
See the check_retry parameter documentation for more information. - Checking file size after transfer with the check-size parameter.
If more than one file is to be transferred then the use of the transaction parameter is recommended (TBD - LINK)
File Completeness - MD5 Hash Checksums
Example a) - Create and transfer an MD5 checksum file
JADE can write a file's MD5 checksum into a file which it gives the ending .MD5. This file can then be transferred along with the 'original' file.
However, to do this, JADE requires write permissions for the source directory. In turn, this means that this feature cannot be demonstrated by downloading files from our test server, where the demo user only has read permissions. Instead it will be demonstrated through local transfer.
[local_2_local_create_md5]
;;
operation = copy
file_spec = ^test_[0-9]\.txt$
;;
CreateSecurityHashFile = true
;;
source_host = localhost
source_protocol = local
source_dir = ${USERPROFILE}\jade_demo\a
;;
target_host = localhost
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
Run the example using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="local_2_local_create_md5"
Behavior
If you have already run some of the previous examples, you will have a series of files in the a folder on your computer, sequentially named test_1.txt through to test_5.txt.
JADE will carry out the following steps when executing the local_2_local_create_md5 profile:
- Save the MD5 hash for each 'original' file it finds matching the file_spec parameter regex in the source directory in a temporary file.
- Transfer the 'original' files to the target directory.
- Copy the files containing the MD5 hash tags to the target directory after transfer of the 'main' files has been completed.
An MD5 file corresponding to a file test_1.txt would carry the name test_1.txt.MD5.
The next example shows the use of the MD5 file to verify the transferred files.
Example b) - Checking a transferred file against a transferred MD5 checksum file
After you have run the Create and transfer an MD5 checksum file example above you will have a series of files in your target folder together with corresponding .MD5 files.
When executing the profile in this example JADE will verify the files that were transferred to the b directory in the previous example using their associated MD5 hash files.
However, to test whether the checks actually work, you can open one of the transferred files and modify part of its content before running the profile below.
[local_2_local_check_md5]
;;
operation = copy
file_spec = ^test_[0-9]\.txt$
;;
CheckSecurityHash = true
;;
source_host = localhost
source_protocol = local
source_dir = ${USERPROFILE}\jade_demo\b
;;
target_host = localhost
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b\checked
Run the example using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="local_2_local_check_md5"
Behavior
First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.
If all of the 'original' files in the source directory b are intact (i.e. if you do not manipulate any of the files) then:
- an MD5 hash will be recalculated for each of the 'original' files.
- These new hashes will then checked against those saved in the .MD5 files.
If all the hashes are OK then:
- An INFO message such as that listed below will be generated for each file:
143 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileList - Security checking: File C:\temp\jade_demo\b\checked/test_1.txt, hash read '80d7b9824ceab0c d1d7d11b3c65a0e87', hash calculated '80d7b9824ceab0cd1d7d11b3c65a0e87'
However, if any of the 'original' files in the source directory b has been corrupted (or if you have manipulated one or more of the files as suggested above) then an error will be thrown.
The transferred files will not be deleted from the target directory - in this case the checked directory.
File Completeness - Atomic File Transfer
Example - Atomic File Transfer
The principle of atomic file transfer was described at the beginning of this article.
The local_2_local_atomic profile listed below is will carry out atomic file transfer from our test server to a local folder. As the transfer is likely to occur within one or two seconds it is unlikely that it will be possible for the functioning of the atomic transfer (i.e the removal of atomic prefix and suffixes) to be observed.
[sftp_server_2_local_atomic]
;;
operation = copy
file_spec = ^test_large_1\.txt$
;;
atomic_prefix = ~
atomic_suffix = ~
;;
source_host = test.sos-berlin.com
source_protocol = sftp
source_ssh_auth_method = password
source_port = 22
source_user = demo
source_password = demo
source_dir = /large/
;;
target_host = localhost
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\a
The Atomic Transfer profile is started using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="sftp_server_2_local_atomic"
Behavior - Atomic File Transfer
When atomic file transfer is specified JADE will add the characters specified in the atomic_prefix and atomic_suffix parameters to the file name. Once the file transfer has been completed, JADE deletes the prefix and suffix.
Whilst this procedure will likely happen to quickly to be observed, the following INFO entries should be able to be seen in the command line interface:
20218 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileListEntry - SOSVfs_I_0108: transfer of /test_large_1.txt started 25943 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileListEntry - SOSVfs_I_274: The security hash (MD5) of the test_large_1.txt file is 40f38a5f677aa387a057677e6d78aaa6 25949 [main] INFO com.sos.VirtualFileSystem.local.SOSVfsLocalFile - SOSVfs_I_150: File 'C:\temp\jade_demo\a\~test_large_1.txt~' renamed to 'C:\temp\jade_demo\a/test_large_1.txt'.
File Completeness - Check Steady State
Example - Check Steady State of incoming files
The local_2_local_check_steady_state profile listed below is intended to be used in JADE together with a second file transfer program that allows the file transfer rate to be restricted. This second file transfer program is used to transfer a large file from our test server to the a folder used in previous examples at a relatively slow rate. In parallel, JADE is used to check the size of the file arriving in the a folder. When a steady state is found, the file will be transferred to the local folder b.
WinSCP is an example of a windows file transfer program that allows transfer rate restriction and was used to test this example.
Carry out the following steps to run the example:
- Prepare your command line interface to run JADE with the local_2_local_check_steady_state profile using the command line call listed below but do not yet start JADE.
- The second file transfer program should be configured to allow a maximum rate of around 16 kB/s. This transfer rate will mean that the download of the example file (approx. 220kB ) will take around 15 seconds. This will provide sufficient time for JADE to be started with the local_2_local_check_steady_state profile below, find the file being transferred and note that its own file transfer operation cannot be started until the download has been completed.
- Log onto our test server with the second file transfer program using the same host, user & password as listed in other examples on this page.
- Select the test_large_1.txt file, which you will find in the demo user's large folder.
- Start downloading the selected file to the jade_demo/a folder.
- Quickly start JADE with the local_2_local_check_steady_state' profile.
[local_2_local_check_steady_state]
;;
operation = copy
file_spec = ^test_large_1\.txt$
check_steady_state_of_files = true
steady_state_count = 10
check_steady_state_interval = 2
;;
source_host = localhost
source_protocol = local
source_dir = ${USERPROFILE}\jade_demo\a
;;
target_host = localhost
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
Start the profile using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="local_2_local_check_steady_state"
Behavior - Check Steady State of incoming files
Once JADE starts executing the profile the command line interface will show a series of messages containing the following:
- 1 file found for the regexp ^test_large_1\.txt$
- Checking file(s) for steady state
- File C:\temp\jade_demo\a\test_large_1\.txt changed during checking steady state (will likely be repeated)
- File C:\temp\jade_demo\a\test_large_1\.txt was not changed during checking steady state
- Transfer of file C:\temp\jade_demo\a\test_large_1\.txt started