Proposals for next features
Large File Handling
Starting Situation
- JADE transfers multiple files in parallel.
- Currently one file is being processed by one channel at a time.
Desired Behavior
- Concerning the transfer of large files (> 1GB) the performance when using one channel for a file is too slow.
- Mutliple channels should be used.
Proposed Solution
- Using multiple channels would increase the overall speed of transfer
- Use of multiple TCP streams to transfer data in parallel
- See also
- http://monalisa.cern.ch/FDT/documentation_fdt.html implements the use of multiple channels
Resume File Transfer
Starting Situation
- If an error occurs during transfer of a file then this file has to be re-transferred completely.
Desired Behavior
- Automatically detect and resume the transfer after interruptions (e.g. connection is broken).
- Detect
- JADE knows when a transfer was interrupted and would resume operation with the same parameters.
- Detection is limited for a configurable duration after interrupted file transfers.
- Resume
- JADE knows the chunks that have been transferred and is capable to continue writing chunks.
- Fail-safe operation
- JADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file.
- Detect
Proposed Solution
- Implementation with the JADE Client.
- Use the history of file transfers to check for interrupted transfers.
- Keep track of chunks that have been transferred successfully in the file transfer history.
- Development of the functionality to detect and resume file transfers.
Priority of files to be transferred
Starting Situation
- If a group of files is to be transferred, e.g. files matched by a regular expression, then they are transferred in the order of sequence that is povided from that selection, which is somewhat arbitrary.
Desired Behaviour
- Individual files of a group of files should be prioritized within one transfer session.
Proposed Solution
- This feature could be based on the use of a list of files as input parameter for file transfer.
- By pre-processing (external command or internal rules) this list would be sorted before transfer.
- Multiple file lists could be specified which would result in the fact that the files given in each list would be completely transferred before processing the next list.
- The sequence in which lists are specified would cause JADE to complete the transfer of each list before processing the next list, thus implementing prioritization by use of different lists.
- Use of channels as specified in the Large File Handling feature should be focussed on prioritized files.
- Workaround for the current release: JADE supports to use a list of files as input parameter. Transfers are then performed in the sequence of that list. During pre-processing that list could be created and sorted by priority.
JMS Interface
Starting Situation
- JADE does not dispose of a JMS interface.
Desired Behaviour
- JADE should implement a JMS interface to
- publish messages to subscribers for a file transfer topic that would signal that a file has been transferred.
- subscribe to file transfer topics that would cause JADE to start a file transfer having received specific messages..
- Some use cases are required to elabore the requirements for this feature.
Proposed Solution
- Implementation with Apache Message Queue
- Possibly implemented for JobScheduler that would run the respective JADE jobs based on events as the receipt of messages.
- Detailed specification required.
NFS Support
Starting Situation
- JADE does not provide NFS support.
- Therefore no Unix file systems using this protocol can be accessed directly by JADE.
- Currently the respective file systems have to be mounted to a Unix system to make them accessible to JADE.
Desired Behavior
- Support for NFS Version 2,3,4
- Have JADE access any file system using NFS in the network.
Proposed Solution
- Possible implementation with libraries from Sun (http://java.net/projects/yanfs) or dCache (https://rb.dcache.org/r/). This is not yet decided on.
HTTP, HTTPS as Data Provider
Starting Situation
- This feature is implemented with limits. Missing functionalities are:
- proxy support
- support for directory listings
- This feature is broken in the current JADE release
- Due to the recently introduced mutli-threading this feature currently does not work.
- It will be fixed with the upcoming release Release 1.7.4321 (1.7.4)
Desired Behaviour
- Have proxy support and directory listings implemented.
- Have the HTTP, HTTPS data provider fixed to work as documented.
Proposed Solution
- Implementation based on Apache Commons VFS:
- support for directory listing, authentication.
- no support for write operations is provided.
LDAP Integration
Starting Situation
- No authentication or authorization is available for JADE.
- JADE is mainly used for batch file transfer, therefore hardly any indvidual users are involved that would have to be authenticated and authorized.
Desired Behavior
- JADE should authenticate to an LDAP directory before performing transfers.
- Some use cases would be helpful to get a better idea of the scope of this feature, i.e. if the file transfer itself, its history GUI and/or JOC as the management interface for file transfers should authenticate.
Proposed Solution
- Authentication of JobScheduler components makes use of Apache Shiro. The same would apply to JADE compoents.
- JADE Client authentication with LDAP.
- JADE Background Service History GUI supports LDAP authentication.
- This component will be available with Release 1.8.0
- JOC should support LDAP
- This could be part of a major re-write of JOC that is envisaged for the next months, see JOC - Cockpit
- We are currently in the process of collecting requirements for a new JOC Cockpit.
Publish-Subscribe Model
Starting Situation
- JADE does not support to transfer files to multiple destinations, see Can JADE use mutliple destinations for file transfer?
- JADE does not support the configuration management of routing rules for a publish-subscribe model
Desired Behavior
- JADE should allow 1-to-many file transfers.
- Such file transfers should be based on a publish-subscribe model as proposed with the feature JMS Interface.
- JADE should provide routing capabilities that would be managed centrally.
Proposed Solution
- Publish-subscribe models are often used with ESB systems, therefore no individual development of that model for JADE is intended. Instead, JADE should implement interfaces to popular ESB systems and use the models offered by ESBs, e.g.
- Apache ServiceMix, Mule
- JADE should support an interface to routing components that are typically used for ESBs, e.g.
- Apache Camel
- JADE should not implement its own configuration management for routing rules but make use of configuration items of a central routing component.
Monitoring Interface
Starting Situation
- JADE supports notifications to System Monitors, e.g. Nagios, op5, Zabbix, if used as a JITL Job in JobScheduler.
- Additional System Monitors can easily be integrated should a command line interface be available.
- Other monitoring solutions with more sophisticated interfaces should be supported.
Desired Behavior
- File transfers effected by the JADE Client should be subject to monitoring in the same way as the current JADE JITL job.
- More Monitors such as IBM Tivoli, HP OVO, IBM WS Message, Broker should be supported.
- SNMP should be supported for interfacing with System Monitors, see Feature Proposal - JobScheduler SNMP Support
Proposed Solution
- Start from the feature JobScheduler Monitoring Interface
- add monitoring capabilities for the file transfer history as reported by the JADE Background Service
- Implement interfaces for popular Monitoring Systems
- IBM Tivoli Monitoring: this should be feasible by a client command line tool and therefore be available with the JobScheduler Monitoring Interface.
- HP OVO: to be clarified if a command line tool exists.
- IBM WS Message Broker (MQ): to be clarified if a message queue interface were required or a command line tool were sufficient.
- SNMP Interface
- SNMP could be supported via a separate process that integrates with the JobScheduler Monitoring interface.
- the overall availability of jobs can be reported
- in case of job errors respective messages are sent to an SNMP server.
- SNMP is restricted to predefined messages.
- Though this is perfect for interfacing with all sorts of System Monitors the messages would not be too specific as the were defined in a standard MIB file
- Such messages would include codes as e.g. for "ERROR-xxx File transfer failed". "ERROR-yyy Target host not found", however, they would not include specific information on the respective files, hosts etc.
- Therefore, we agree that this is a desirable feature and at the same time we suggest to use the more versatile JobScheduler Monitoring Interface with System Monitors that would transmit more detailed information as it is required in case of errors in file transfers.
- SNMP could be supported via a separate process that integrates with the JobScheduler Monitoring interface.
Transfer Speed Restrictions
Starting situation
- JADE uses all the available bandwith for a transfer and leaves it to the network administration to manage rules for bandwidth usage.
Desired Behavior
- JADE should be configurable for use of bandwidth, e.g. for use in slow networks.
Proposed Solution
- Technical specification required
File Integrity Checking
Starting Situation
- JADE has the following parameters around the integrity hash
CreateSecurityHash
(default true)CreateSecurityHashFile
(default false)CheckSecurityHash
(default false)SecurityHashType
(default MD5)
- If
CreateSecurityHash=true
then only the integrity hash of the target file is calculated.- If
CheckSecurityHash=false
thenCreateSecurityHashFile=true
has no effect and the integrity hash file is not created.
However, JADE tries to transfer an integrity hash file which doesn't exist and subsequently an error is thrown.
- If
- A check of the integrity hash is not implemented. In particular, the integrity hash of the source file has to be calculated too.
- If
CheckSecurityHash=true
then JADE tries to transfer a security hash file which doesn't necessarily exist and an error is thrown.
- If
- If
CreateSecurityHashFile=true
then the security hash file will be created on the source but- ... it can only be created if the source has the "local" protocol
- ... an error occurred if the source protocol is not "local".
Desired Behavior
- A integrity hash is created for source files and for target files to check if the transfer was complete and accurate.
- Integrity is checked by creating an integrity hash per source file and per target file and by comparing the checksum results.
- The parameter
CreateSecurityHash
will be removed as it is replaced by the parameterCheckSecurityHash
. - If
CheckSecurityHash=true
then the integrity hash of the source and the target file are calculated and will be checked after the transfer.- If the hashes are unequal then the transfer rolls back.
- The existing parameters
SecurityHash
will be renamded toIntegrityHash
.
Proposed Solution
- Feature specification is ready.
Checksum File
Current Situation
- A checksum file with the extension .md5 is created if
- the parameter
CreateSecurityHashFile
is set totrueand
- the parameter
CreateSecurityHash
is set totrue.
- the parameter
- This feature is only available if
- the source protocol
local
is used or - a jump host is being used with the operation
copytointernet
.
- the source protocol
- If the parameter
file_compress
is used then the md5 value in the checksum file is calculated from the non-compressed input file. The checksum file itself is compressed.
Desired Behavior
- If
CreateSecurityHashFile=true
the integrity hash of the target file is stored in a file on the target.
This file has the same name plus the integrity hash type (md5) as extension. - This feature works independent form the setting of the parameter
CreateSecurityHash
. - This feature shall be available independently from the protocol used for source and target systems.
- If the parameter
file_compress
is used then the checksum file itself is not compressed. - If the parameters
replacement
andreplacing
are used then the name of the checksum file is created from the resulting name of the file on the target system. - The existing parameters
SecurityHash
will be renamded toIntegrityHash
.
Proposed Solution
- Feature specification is ready.