Introduction
This is the fifth in a series of articles describing how to get started with using the JADE Client via its Command Line Interface.
JADE Tutorials List
JADE Client Command Line Interface
- Getting Started and Downloading files
- Simple File Selection
- More Advanced File Selection
- Checking files for completeness
- Public / Private Key Authentication
- Transfer via a Jump Host / DMZ
JADE Background Service
Prerequisites
See the Using the tutorials with the JADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.
Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.
Download file
The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor using the following link:
Checking files for completeness
A number of methods for checking whether file transfer has been completed are presented in this document:
- Integrity hash checksums such as MD5, which are generated before file transfer starts and can be used to check files after transfer.
- Atomic file transfer, which is used to hide files being transferred from target directory monitors until the file transfer has been completed.
- A prefix and/or suffix is added to the name of the file for the duration of the transfer.
This prefix/suffix is chosen so that the file being transferred will not be matched by any target host file monitor. - Once transfer has been completed, JADE removes the prefix/suffix and the file becomes visible to a file monitor.
- A prefix and/or suffix is added to the name of the file for the duration of the transfer.
- Regularly checking the file size until it reaches a steady state. Here it is assumed that the transfer has been completed when two checks in series return the same size.
Other methods are:
- Repeating the file transfer and comparing the sizes of the file transfers. When both transfers have the same size then it is assumed that the transfer is complete.
See the CheckSize element documentation for more information. - Checking file size after transfer with the CheckSize element.
Use of the transactional parameter is generally recommended if more than one file is to be transferred.
File Completeness - Integrity Hash Checksums
Example a) - Creating an integrity hash file
JADE can generate an MD5 checksum file for a file. This file which is given the ending .md5 and can then be transferred along with the 'original' file. Note however that JADE only generates the checksum file during a file transfer operation, whilst it is reading the 'original' file from the transfer source. This means that this feature has a limited validity and that in practice JADE checks files for completeness using integrity hash files that have been generated by another source.
Note that to create and save an integrity hash file before a file transfer operation, JADE requires write permissions for the source directory. In turn, this means that this feature cannot be demonstrated by downloading files from our test server, where the demo user only has read permissions. Instead it will be demonstrated through local transfer.
Note
- This example uses a set of 5 test_*.txt files that have been saved into the user's \jade_demo\a directory by an earlier tutorial in this series - for example The JADE Client Command Line Interface - Tutorial 1 - Getting Started .
- A ProtocolFragment is not required in the configuration of this example, as local to local transfer is being carried out.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_create_md5" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
Behavior
If you have already run some of the previous examples, you will have a series of files in the ${USERPROFILE}\jade_demo\a directory on your computer, sequentially named test_1.txt through to test_5.txt.
JADE will carry out the following steps when executing the local_2_local_create_md5 Profile:
- Read each of the files in the source directory matching the FileSpec regular expression into memory
- Generate an integrity hash file for each 'original' files it finds matching the FileSpec.
- Save the integrity hash files along with the 'original' files in the transfer target directory (
${USERPROFILE}\jade_demo\b). - An MD5 file corresponding to a file test_1.txt would carry the name test_1.txt.md5.
main INFO 13:57:35,512 (SOSFileListEntry.java:302) ::createChecksumFile SOSVfs_I_285: Checksum-Datei erzeugt: 'C:\Use rs\aa\jade_demo\b\test_1.txt.md5'
The next example shows the use of the integrity hash files to verify files that have been transferred.
Example b) - Checking transferred files against transferred integrity hash files
After you have run the Create and transfer an MD5 checksum file example above you will have a series of files in your target ${USERPROFILE}\jade_demo\b directory together with the corresponding hash files.
When executing the profile in this example JADE will verify the 'original' files that were transferred to the b directory using their associated integrity hash files before writing the 'original' files to a new target directory ${USERPROFILE}\jade_demo\b\checked.
However, to test whether the checks actually work, you can open one of the transferred files and modify part of its content before running the profile below.
XML Editor Configuration
The local_2_local_check_md5 profile shown in the screenshot below is almost identical to the previous local_2_local_create_md5 profile - the only significant difference is that it does not have the CreateIntegrityHashFile parameter as the files are already there.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
Behavior
First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.
If all of the 'original' files in the source directory b are intact (i.e. if you do not manipulate any of the files) then:
- a new integrity hash will be calculated for each of the 'original' files.
- These new hashes will then checked against those saved in the .md5 files.
If all the hashes are OK then:
- No information about the results of the integrity checks will be output.
- See Issue
Behavior in the Event of an Integrity Hash Mismatch
If any of the 'original' files in the source directory b has been corrupted (or if you have manipulated one or more of the files as suggested above) then an error will be thrown:
main ERROR 14:11:13,763 (SOSFileListEntry.java:1027) ::run SOSVfs_E_229: Fehler. Daten³bertragung nicht m÷glich. Grund: com.sos.JSHelper.Exceptions.JobSchedulerException: Integrity Hash violation. File C:\Users\aa\jade_demo\b\test_3.txt.md5, checksum read: 'b4817065c96c3103e11ab8295a8e0c2c', checksum calculated: 'dbeb552a55aad26cabf17641104bf64f' main ERROR 14:11:13,765 (SOSDataExchangeEngine.java:1275) ::transfer SOSDataExchangeEngine.TRANSFER_ABORTED main INFO 14:11:13,766 (SOSFileList.java:506) ::Rollback Rollback aborted files.
Note that:
- Files that have already been checked and fully transferred will only be deleted from the target directory - in this case the checked directory - if transactional transfer has been specified.
- File transfers that have not been completed / started when the error occurs will be aborted / rolled back.
File Completeness - Atomic File Transfer
Example - Atomic File Transfer
The principle of atomic file transfer was described at the beginning of this article.
The local_2_local_atomic profile listed below is will carry out atomic file transfer from our test server to a local folder. As the transfer is likely to occur within one or two seconds it is unlikely that it will be possible for the functioning of the atomic transfer (i.e the removal of atomic prefix and suffixes) to be observed.
[sftp_server_2_local_atomic]
;;
operation = copy
file_spec = ^test_large_1\.txt$
;;
atomic_prefix = ~
atomic_suffix = ~
;;
source_host = test.sos-berlin.com
source_protocol = sftp
source_ssh_auth_method = password
source_port = 22
source_user = demo
source_password = demo
source_dir = /large/
;;
target_host = localhost
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\a
The Atomic Transfer profile is started using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="sftp_server_2_local_atomic"
Behavior - Atomic File Transfer
When atomic file transfer is specified JADE will add the characters specified in the atomic_prefix and atomic_suffix parameters to the file name. Once the file transfer has been completed, JADE deletes the prefix and suffix.
Whilst this procedure will likely happen to quickly to be observed, the following INFO entries should be able to be seen in the command line interface:
20218 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileListEntry - SOSVfs_I_0108: transfer of /test_large_1.txt started 25943 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileListEntry - SOSVfs_I_274: The security hash (MD5) of the test_large_1.txt file is 40f38a5f677aa387a057677e6d78aaa6 25949 [main] INFO com.sos.VirtualFileSystem.local.SOSVfsLocalFile - SOSVfs_I_150: File 'C:\temp\jade_demo\a\~test_large_1.txt~' renamed to 'C:\temp\jade_demo\a/test_large_1.txt'.
File Completeness - Check Steady State
Example - Check Steady State of incoming files
The local_2_local_check_steady_state profile listed below is intended to be used in JADE together with a second file transfer program that allows the file transfer rate to be restricted. This second file transfer program is used to transfer a large file from our test server to the a folder used in previous examples at a relatively slow rate. In parallel, JADE is used to check the size of the file arriving in the a folder. When a steady state is found, the file will be transferred to the local folder b.
WinSCP is an example of a windows file transfer program that allows transfer rate restriction and was used to test this example.
Carry out the following steps to run the example:
- Prepare your command line interface to run JADE with the local_2_local_check_steady_state profile using the command line call listed below but do not yet start JADE.
- The second file transfer program should be configured to allow a maximum rate of around 16 kB/s. This transfer rate will mean that the download of the example file (approx. 220kB ) will take around 15 seconds. This will provide sufficient time for JADE to be started with the local_2_local_check_steady_state profile below, find the file being transferred and note that its own file transfer operation cannot be started until the download has been completed.
- Log onto our test server with the second file transfer program using the same host, user & password as listed in other examples on this page.
- Select the test_large_1.txt file, which you will find in the demo user's large folder.
- Start downloading the selected file to the jade_demo/a folder.
- Quickly start JADE with the local_2_local_check_steady_state' profile.
[local_2_local_check_steady_state]
;;
operation = copy
file_spec = ^test_large_1\.txt$
check_steady_state_of_files = true
steady_state_count = 10
check_steady_state_interval = 2
;;
source_host = localhost
source_protocol = local
source_dir = ${USERPROFILE}\jade_demo\a
;;
target_host = localhost
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
Start the profile using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="local_2_local_check_steady_state"
Behavior - Check Steady State of incoming files
Once JADE starts executing the profile the command line interface will show a series of messages containing the following:
- 1 file found for the regexp ^test_large_1\.txt$
- Checking file(s) for steady state
- File C:\temp\jade_demo\a\test_large_1\.txt changed during checking steady state (will likely be repeated)
- File C:\temp\jade_demo\a\test_large_1\.txt was not changed during checking steady state
- Transfer of file C:\temp\jade_demo\a\test_large_1\.txt started