Propositions for next features
Large File Handling
Problem
- Concerns the transfer of large files (> 1GB)
- Performance is too slow
Proposed Solution
- Currently one file is being processed in one channel.
- Using multiple channels would increase the overall speed of transfer
- Use of multiple TCP streams to transfer data in parallel
- See also
- http://monalisa.cern.ch/FDT/documentation_fdt.html implements the use of multiple channels
Resume File Transfer
Problem
- If an error occurs during transfer of a file then this file has to re-transferred completely
Proposed Solution
- Automatically detect and resume the transfer after interruptions (e.g. connection is broken)
- Detect
- JADE knows when a transfer was interrupted and would resume operation with the same parameters
- Detection is limited for a configurable duration after interrupted file transfers
- Resume
- JADE knows the chunks that have been transferred and is capable to continue writing chunks
- Fail-safe operation
- JADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file
- Detect
Priority of files to be transferred
Problem
- Individual files of a file group should be prioritized within one transfer session.
Solution
- This feature will be based on the use of a list of files as input parameter for file transfer.
- By pre-processing (external command or internal rules) this list would be sorted before transfer.
- Multiple file lists could be specified which would result in the fact that the files of each list would be completely transferred before processing the next list.
- Use of channels as specified in the Large File Handling features should be focussed on prioritized files.
- Workaround: JADE supports to use a list of files as input parameter. Transfers are then performed in the sequence of that list. During pre-processing that list could be created and sorted by priority.
JMS Interface
Problem
- JADE does not dispose of a JMS interface
Proposed Solution
- Implementation with Apache Message Queue
- Specification required
NFS Interface
Problem
- NFS is not available for JADE
Proposed Solution
- Support NFS Version 2,3,4
- Possible implementation with libraries from Sun (http://java.net/projects/yanfs) or dCache (https://rb.dcache.org/r/)
HTTP, HTTPS as Data Provider
Problem
- This feature is implemented with limits. Missing functionalities are:
- proxy support
- support for directory listings
- This feature is broken in the current JADE release
- Due to the recently introduced mutli-threading this feature is does not work
Proposed Solution
- Implementation based on Apache Commons VFS
- support for directory listing, authentication
- no support for write operations
Integration with LDAP
Problem
- No authentication or authorization is available for JADE
Proposed Solution
- JADE Background Service History GUI supports LDAP authentication.
- This component will be available with release 1.8
- JOC should support LDAP
- This could be part of a major re-write of JOC that will take some months
- tbd: das problem ist nicht so schwerwiegend, da hier keine Enduser im Spiel sind
Publish-Subscribe Model
Problem
- JADE does not support to transfer files to multiple destinations, see Can JADE use mutliple destinations for file transfer?
- JADE does not support the configuration management of routing rules for a publish-subscribe model
Solution
- Publish-Subscribe models are often used with ESB systems, therefore no individual development for JADE is intended. Instead, JADE should support interface to popular ESB systems.
- Apache ServiceMix, Mule
- JADE should support interface to routing components that are used for ESBs
- Apache Camel
- JADE should not implement its own configuration management for routing rules
- (Graphik erstellen)
Monitoring Interfaces
Problem
- JADE supports notifications to System Monitors, e.g. Nagios, op5, Zabbix, if used as a JITL Job in JobScheduler
- Additional System Monitors can easily be integrated should a command line interface be available.
- Other monitoring solutions with more sophisticated interfaces should be supported.
Proposed Solution
- Start from the existing JobScheduler Monitoring Interface
- add monitoring capabilities for the file transfer history as reported by the JADE Background Service
- Implement interfaces for popular Monitoring Systems
- IBM Tivoli Monitoring: should this be feasible by a client command line tool then
- HP OVO: tbd
- IBM WS Message Broker (MQ)
- SNMP Interface: tbd
- architecture
Further Propositions
Transfer speed restrictions
Problem
- JADE uses all the available bandwith for a transfer and leaves it to the network administration to manage rules for bandwidth usage
- JADE should be configurable for use of bandwidth, e.g. in slow networks
Proposed Solution
- tbd