...
File Completeness - Integrity Hash Checksums
Example a) - Creating an
...
integrity hash file
JADE can generate an MD5 checksum file for a file. This file which is given the ending .md5 and can then be transferred along with the 'original' file. Note however that JADE only generates the checksum file during a file transfer operation, whilst it is reading the 'original' file from the transfer source. This means that this feature has a limited validity and that in practice JADE checks files for completeness using integrity hash files that have been generated by another source.
...
- This example uses a set of 5 test_*.txt files that have been saved into the user's \jade_demo\a directory by an earlier tutorial in this series - for example The JADE Client Command Line Interface - Tutorial 1 - Getting Started .
- A ProtocolFragment is not required in the configuration of this example, as local to local transfer is being carried out.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
...
If you have already run some of the previous examples, you will have a series of files in the ${USERPROFILE}\jade_demo\a folder directory on your computer, sequentially named test_1.txt through to test_5.txt.
...
- Read each of the files in the source directory matching the FileSpec regular expression into memory
- Generate an integrity hash file for each 'original' files it finds matching the FileSpec.
- Save the integrity hash files along with the 'original' files in the transfer target directory (
${USERPROFILE}\jade_demo\b). - An MD5 file corresponding to a file test_1.txt would carry the name test_1.txt.md5.
The next example shows the use of the MD5 file integrity hash files to verify the files that have been transferred files.
Example b) - Checking
...
transferred
...
files against
...
transferred integrity hash files
After you have run the Create and transfer an MD5 checksum file example above you will have a series of files in your target folder target ${USERPROFILE}\jade_demo\b directory together with the corresponding .MD5 hash files.
When executing the profile in this example JADE will verify the 'original' files that were transferred to the b directory in the previous example using their associated MD5 hash files. integrity hash files before writing the 'original' files to a new target directory ${USERPROFILE}\jade_demo\b\checked.
However, to test whether the checks actually work, you can open one of the transferred files and modify part of its content before running the profile below.
...
| language | bash |
|---|
XML Editor Configuration
...
The
local_2_local_check_md5 profile shown in the screenshot below is almost identical to the previous local_2_local_create_md5 profile - the only significant difference is that it does not have the CreateIntegrityHashFile parameter.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
<?xml version="1.0" encoding="utf-8"?> <Configurations> <Fragments> <ProtocolFragments /> </Fragments> <Profiles> <Profile profile_id="local_2_local_check_md5"> <Operation> <Copy> ] ;; operation = copy file_spec = ^test_[0-9]\.txt$ ;; check_security_hash = true ;; source_host = localhost source_protocol = local<CopySource> source_dir <CopySourceFragmentRef> = ${USERPROFILE}\jade_demo\b ;; target_host <LocalSource /> </CopySourceFragmentRef> <SourceFileOptions> = localhost<Selection> target_protocol = local<FileSpecSelection> target_dir = ${USERPROFILE}\jade_demo\b\checked |
...
<FileSpec><![CDATA[^(test)(_)[0-9]\.txt]]></FileSpec>
<Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory>
</FileSpecSelection>
</Selection>
</SourceFileOptions>
</CopySource>
<CopyTarget>
<CopyTargetFragmentRef>
<LocalTarget />
</CopyTargetFragmentRef>
<Directory><![CDATA[${USERPROFILE}\jade_demo\b]]></Directory>
<TargetFileOptions>
<CheckIntegrityHash />
</TargetFileOptions>
</CopyTarget>
</Copy>
</Operation>
</Profile>
</Profiles>
</Configurations> |
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
[local_2_local_create_md5]
operation = copy
source_protocol = local
file_spec = ^(test)(_)[0-9]\.txt
source_dir = ${USERPROFILE}\jade_demo\a
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
check_security_hash = true
create_security_hash_file = true
[local_2_local_check_md5]
operation = copy
source_protocol = local
file_spec = ^(test)(_)[0-9]\.txt
source_dir = ${USERPROFILE}\jade_demo\a
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
check_security_hash = true |
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
| Code Block | ||||
|---|---|---|---|---|
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5" |
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
| Code Block | ||||
|---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5" |
Behavior
First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.
If all of the 'original' files in the source directory b are intact (i.e. if you do not manipulate any of the files) then:
- a new integrity hash will be calculated for each of the 'original' files.
- These new hashes will then checked against those saved in the .md5 files.
If all the hashes are OK then:
- An INFO message such as that listed below will be generated for each file:
| Code Block | ||
|---|---|---|
| ||
jade.cmd -settings="%USERPROFILE%143 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileList - Security checking: File C:\temp\jade_demo\b\jadechecked/test_settings.ini" -profile="local_2_local_check_md5" |
Behavior
First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.
1.txt, hash read '80d7b9824ceab0c
d1d7d11b3c65a0e87', hash calculated '80d7b9824ceab0cd1d7d11b3c65a0e87' |
Behavior in the Event of an Integrity Hash Mismatch
If any If all of the 'original' files in the source directory b are intact (i.e. if you do not manipulate any has been corrupted (or if you have manipulated one or more of the files as suggested above) then :
- an MD5 hash will be recalculated for each of the 'original' files.
- These new hashes will then checked against those saved in the .MD5 files.
If all the hashes are OK then:
...
an error will be thrown:
| Code Block | ||
|---|---|---|
| ||
143 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileList - Security checking:
File C:\temp\jade_demo\b\checked/test_1.txt, hash read '80d7b9824ceab0c
d1d7d11b3c65a0e87', hash calculated '80d7b9824ceab0cd1d7d11b3c65a0e87' |
However, if any of the 'original' files in the source directory b has been corrupted (or if you have manipulated one or more of the files as suggested above) then an error will be thrown.
...
| |||
main ERROR 14:11:13,763 (SOSFileListEntry.java:1027) ::run SOSVfs_E_229: Fehler. Daten³bertragung nicht m÷glich. Grund: com.sos.JSHelper.Exceptions.JobSchedulerException: Integrity Hash violation. File C:\Users\aa\jade_demo\b\test_3.txt.md5, checksum read: 'b4817065c96c3103e11ab8295a8e0c2c', checksum calculated: 'dbeb552a55aad26cabf17641104bf64f'
main ERROR 14:11:13,765 (SOSDataExchangeEngine.java:1275) ::transfer SOSDataExchangeEngine.TRANSFER_ABORTED
main INFO 14:11:13,766 (SOSFileList.java:506) ::Rollback Rollback aborted files. |
Note that:
- Files that have already been checked and fully transferred will only be deleted from the target directory - in this case the checked directory - if transactional transfer has been specified.
- File transfers that have not been completed / started when the error occurs will be aborted / rolled back.
File Completeness - Atomic File Transfer
...