Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Propositions for next features

Large File Handling

Starting Situation

  • JADE transfers multiple files in parallel.
  • Currently one file is being processed by one channel at a time.

Desired Behavior

  • Concerning the transfer of large files (> 1GB) the performance when using one channel for a file is too slow.
  • Mutliple channels should be used.

Proposed Solution

Resume File Transfer

Starting Situation

  • If an error occurs during transfer of a file then this file has to be re-transferred completely.

Desired Behavior

  • Automatically detect and resume the transfer after interruptions (e.g. connection is broken).
    • Detect 
      • JADE knows when a transfer was interrupted and would resume operation with the same parameters.
      • Detection is limited for a configurable duration after interrupted file transfers.
    • Resume
      • JADE knows the chunks that have been transferred and is capable to continue writing chunks.
    • Fail-safe operation
      • JADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file.

Proposed Solution

  • Implementation with the JADE Client.
  • Use the history of file transfers to check for interrupted transfers.
  • Keep track of chunks that have been transferred successfully in the file transfer history.
  • Development of the functionality to detect and resume file transfers.

Priority of files to be transferred

Starting Situation

  • If a group of files is to be transferred, e.g. files matched by a regular expression, then they are transferred in the order of sequence that is povided from that selection, which is somewhat arbitrary.

Desired Behaviour

  • Individual files of a group of files should be prioritized within one transfer session.

Proposed Solution

  • This feature could be based on the use of a list of files as input parameter for file transfer. 
    • By pre-processing (external command or internal rules) this list would be sorted before transfer.
    • Multiple file lists could be specified which would result in the fact that the files given in each list would be completely transferred before processing the next list.
    • The sequence in which lists are specified would cause JADE to complete the transfer of each list before processing the next list, thus implementing prioritization by use of different lists.
    • Use of channels as specified in the Large File Handling feature should be focussed on prioritized files.
  • Workaround for the current release: JADE supports to use a list of files as input parameter. Transfers are then performed in the sequence of that list. During pre-processing that list could be created and sorted by priority.

JMS Interface

Starting Situation

  • JADE does not dispose of a JMS interface.

Desired Behaviour

  • JADE should implement a JMS interface to
    • publish messages to subscribers for a file transfer topic that would signal that a file has been transferred.
    • subscribe to file transfer topics that would cause JADE to start a file transfer having received specific messages..
  • Some use cases are required to elabore the requirements for this feature.

Proposed Solution

  • Implementation with Apache Message Queue
  • Possibly implemented for JobScheduler that would run the respective JADE jobs based on events as the receipt of messages.
  • Detailed specification required.

NFS Support

Starting Situation 

  • JADE does not provide NFS support. 
  • Therefore no Unix file systems using this protocol can be accessed directly by JADE.
  • Currently the respective file systems have to be mounted to a Unix system to make them accessible to JADE.

Desired Behavior

  • Support for NFS Version 2,3,4
  • Have JADE access any file system using NFS in the network.

Proposed Solution

HTTP, HTTPS as Data Provider

Starting Situation

  • This feature is implemented with limits. Missing functionalities are:
    • proxy support
    • support for directory listings
  • This feature is broken in the current JADE release 

Desired Behaviour

  • Have proxy support and directory listings implemented.
  • Have the HTTP, HTTPS data provider fixed to work as documented.

Proposed Solution

  • Implementation based on Apache Commons VFS:
    • support for directory listing, authentication.
    • no support for write operations is provided.

LDAP Integration

Starting Situation

  • No authentication or authorization is available for JADE.
  • JADE is mainly used for batch file transfer, therefore hardly any indvidual users are involved that would have to be authenticated and authorized.

Desired Behavior

  • JADE should authenticate to an LDAP directory before performing transfers.
  • Some use cases would be helpful to get a better idea of the scope of this feature, i.e. if the file transfer itself, its history GUI and/or JOC as the management interface for file transfers should authenticate.

Proposed Solution

  • Authentication of JobScheduler components makes use of Apache Shiro. The same would apply to JADE compoents.
  • JADE Client authentication with LDAP. 
  • JADE Background Service History GUI supports LDAP authentication.
  • JOC should support LDAP
    • This could be part of a major re-write of JOC that is envisaged for the next months, see JOC - Cockpit
    • We are currently in the process of collecting requirements for a new JOC Cockpit.

Publish-Subscribe Model

Starting Situation

Desired Behavior

  • JADE should allow 1-to-many file transfers.
  • Such file transfers should be based on a publish-subscribe model as proposed with the feature JMS Interface.
  • JADE should provide routing capabilities that would be managed centrally.

Proposed Solution

  • Publish-subscribe models are often used with ESB systems, therefore no individual development of that model for JADE is intended. Instead, JADE should implement interfaces to popular ESB systems and use the models offered by ESBs, e.g.
    • Apache ServiceMix, Mule
  •  JADE should support an interface to routing components that are typically used for ESBs, e.g.
    • Apache Camel
  • JADE should not implement its own configuration management for routing rules but make use of configuration items of a central routing component.

Monitoring Interface

Starting Situation

  • JADE supports notifications to System Monitors, e.g. Nagios, op5, Zabbix, if used as a JITL Job in JobScheduler.
  • Additional System Monitors can easily be integrated should a command line interface be available.
  • Other monitoring solutions with more sophisticated interfaces should be supported.

Desired Behavior

  • File transfers effected by the JADE Client should be subject to monitoring in the same way as the current JADE JITL job.
  • More Monitors such as 

Proposed Solution

  • Start from the feature JobScheduler Monitoring Interface
    • add monitoring capabilities for the file transfer history as reported by the JADE Background Service
  • Implement interfaces for popular Monitoring Systems
    • IBM Tivoli Monitoring: this should be feasible by a client command line tool and therefore be available with the JobScheduler Monitoring Interface.
    • HP OVO: to be clarified if a command line tool exists.
    • IBM WS Message Broker (MQ): to be clarified if a message queue interface were required or a command line tool were sufficient.
  • SNMP Interface
    • architecture

Further Propositions

Transfer speed restrictions

Problem

  • JADE uses all the available bandwith for a transfer and leaves it to the network administration to manage rules for bandwidth usage
  • JADE should be configurable for use of bandwidth, e.g. in slow networks

Proposed Solution

  • tbd

 

 

 

  • No labels