Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 21 Next »

Proposals for next features

Large File Handling

Starting Situation

  • JADE transfers multiple files in parallel.
  • Currently one file is being processed by one channel at a time.

Desired Behavior

  • Concerning the transfer of large files (> 1GB) the performance when using one channel for a file is too slow.
  • Mutliple channels should be used.

Proposed Solution

Resume File Transfer

Starting Situation

  • If an error occurs during transfer of a file then this file has to be re-transferred completely.

Desired Behavior

  • Automatically detect and resume the transfer after interruptions (e.g. connection is broken).
    • Detect 
      • JADE knows when a transfer was interrupted and would resume operation with the same parameters.
      • Detection is limited for a configurable duration after interrupted file transfers.
    • Resume
      • JADE knows the chunks that have been transferred and is capable to continue writing chunks.
    • Fail-safe operation
      • JADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file.

Proposed Solution

  • Implementation with the JADE Client.
  • Use the history of file transfers to check for interrupted transfers.
  • Keep track of chunks that have been transferred successfully in the file transfer history.
  • Development of the functionality to detect and resume file transfers.

Priority of files to be transferred

Starting Situation

  • If a group of files is to be transferred, e.g. files matched by a regular expression, then they are transferred in the order of sequence that is povided from that selection, which is somewhat arbitrary.

Desired Behaviour

  • Individual files of a group of files should be prioritized within one transfer session.

Proposed Solution

  • This feature could be based on the use of a list of files as input parameter for file transfer. 
    • By pre-processing (external command or internal rules) this list would be sorted before transfer.
    • Multiple file lists could be specified which would result in the fact that the files given in each list would be completely transferred before processing the next list.
    • The sequence in which lists are specified would cause JADE to complete the transfer of each list before processing the next list, thus implementing prioritization by use of different lists.
    • Use of channels as specified in the Large File Handling feature should be focussed on prioritized files.
  • Workaround for the current release: JADE supports to use a list of files as input parameter. Transfers are then performed in the sequence of that list. During pre-processing that list could be created and sorted by priority.

JMS Interface

Starting Situation

  • JADE does not dispose of a JMS interface.

Desired Behaviour

  • JADE should implement a JMS interface to
    • publish messages to subscribers for a file transfer topic that would signal that a file has been transferred.
    • subscribe to file transfer topics that would cause JADE to start a file transfer having received specific messages..
  • Some use cases are required to elabore the requirements for this feature.

Proposed Solution

  • Implementation with Apache Message Queue
  • Possibly implemented for JobScheduler that would run the respective JADE jobs based on events as the receipt of messages.
  • Detailed specification required.

NFS Support

Starting Situation 

  • JADE does not provide NFS support. 
  • Therefore no Unix file systems using this protocol can be accessed directly by JADE.
  • Currently the respective file systems have to be mounted to a Unix system to make them accessible to JADE.

Desired Behavior

  • Support for NFS Version 2,3,4
  • Have JADE access any file system using NFS in the network.

Proposed Solution

HTTP, HTTPS as Data Provider

Starting Situation

  • This feature is implemented with limits. Missing functionalities are:
    • proxy support
    • support for directory listings
  • This feature is broken in the current JADE release 
    • Due to the recently introduced mutli-threading this feature currently does not work.
    • It will be fixed with the upcoming release Release 1.7.4321 (1.7.4)

Desired Behaviour

  • Have proxy support and directory listings implemented.
  • Have the HTTP, HTTPS data provider fixed to work as documented.

Proposed Solution

  • Implementation based on Apache Commons VFS:
    • support for directory listing, authentication.
    • no support for write operations is provided.

LDAP Integration

Starting Situation

  • No authentication or authorization is available for JADE.
  • JADE is mainly used for batch file transfer, therefore hardly any indvidual users are involved that would have to be authenticated and authorized.

Desired Behavior

  • JADE should authenticate to an LDAP directory before performing transfers.
  • Some use cases would be helpful to get a better idea of the scope of this feature, i.e. if the file transfer itself, its history GUI and/or JOC as the management interface for file transfers should authenticate.

Proposed Solution

  • Authentication of JobScheduler components makes use of Apache Shiro. The same would apply to JADE compoents.
  • JADE Client authentication with LDAP. 
  • JADE Background Service History GUI supports LDAP authentication.
  • JOC should support LDAP
    • This could be part of a major re-write of JOC that is envisaged for the next months, see JOC - Cockpit
    • We are currently in the process of collecting requirements for a new JOC Cockpit.

Publish-Subscribe Model

Starting Situation

Desired Behavior

  • JADE should allow 1-to-many file transfers.
  • Such file transfers should be based on a publish-subscribe model as proposed with the feature JMS Interface.
  • JADE should provide routing capabilities that would be managed centrally.

Proposed Solution

  • Publish-subscribe models are often used with ESB systems, therefore no individual development of that model for JADE is intended. Instead, JADE should implement interfaces to popular ESB systems and use the models offered by ESBs, e.g.
    • Apache ServiceMix, Mule
  •  JADE should support an interface to routing components that are typically used for ESBs, e.g.
    • Apache Camel
  • JADE should not implement its own configuration management for routing rules but make use of configuration items of a central routing component.

Monitoring Interface

Starting Situation

  • JADE supports notifications to System Monitors, e.g. Nagios, op5, Zabbix, if used as a JITL Job in JobScheduler.
  • Additional System Monitors can easily be integrated should a command line interface be available.
  • Other monitoring solutions with more sophisticated interfaces should be supported.

Desired Behavior

  • File transfers effected by the JADE Client should be subject to monitoring in the same way as the current JADE JITL job.
  • More Monitors such as IBM Tivoli, HP OVO, IBM WS Message, Broker should be supported.
  • SNMP should be supported for interfacing with System Monitors, see Feature Proposal - JobScheduler SNMP Support

Proposed Solution

  • Start from the feature JobScheduler Monitoring Interface
    • add monitoring capabilities for the file transfer history as reported by the JADE Background Service
  • Implement interfaces for popular Monitoring Systems
    • IBM Tivoli Monitoring: this should be feasible by a client command line tool and therefore be available with the JobScheduler Monitoring Interface.
    • HP OVO: to be clarified if a command line tool exists.
    • IBM WS Message Broker (MQ): to be clarified if a message queue interface were required or a command line tool were sufficient.
  • SNMP Interface
    • SNMP could be supported via a separate process that integrates with the JobScheduler Monitoring interface.
      • the overall availability of jobs can be reported
      • in case of job errors respective messages are sent to an SNMP server.
    • SNMP is restricted to predefined messages. 
      • Though this is perfect for interfacing with all sorts of System Monitors the messages would not be too specific as the were defined in a standard MIB file
      • Such messages would include codes as e.g. for "ERROR-xxx File transfer failed". "ERROR-yyy Target host not found", however, they would not include specific information on the respective files, hosts etc.
    • Therefore, we agree that this is a desirable feature and at the same time we suggest to use the more versatile JobScheduler Monitoring Interface with System Monitors that would transmit more detailed information as it is required in case of errors in file transfers.

Transfer Speed Restrictions

Starting situation

  • JADE uses all the available bandwith for a transfer and leaves it to the network administration to manage rules for bandwidth usage.

Desired Behavior

  • JADE should be configurable for use of bandwidth, e.g. for use in slow networks.

Proposed Solution

  • Technical specification required

File Integrity Checking

Starting Situation

  • JADE has the following parameters around the integrity hash
    • CreateSecurityHash (default true)
    • CreateSecurityHashFile (default false)
    • CheckSecurityHash (default false)
    • SecurityHashType (default MD5)
  • If CreateSecurityHash=true then only the integrity hash of the target file is calculated.
    • If CheckSecurityHash=false then CreateSecurityHashFile=true has no effect and the integrity hash file is not created.
      However, JADE tries to transfer an integrity hash file which doesn't exist and subsequently an error is thrown.
  • A check of the integrity hash is not implemented. In particular, the integrity hash of the source file has to be calculated too.
    • If CheckSecurityHash=true then JADE tries to transfer a security hash file which doesn't necessarily exist and an error is thrown.
  • If CreateSecurityHashFile=true then the security hash file will be created on the source but
    • ... it can only be created if the source has the "local" protocol
    • ... an error occurred if the source protocol is not "local".

Desired Behavior

  • A integrity hash is created for source files and for target files (not for intermediate files on a jump host) to check if the transfer was complete and accurate.
  • Integrity is checked by creating an integrity hash per source file and per target file and by comparing the checksum results.
  • The parameter CreateSecurityHash will be removed as it is replaced by the parameter CheckSecurityHash.
  • If CheckSecurityHash=true then the integrity hash of the source and the target file are calculated and will be checked after the transfer.
    • If the hashes are unequal then the transfer rolls back.
  • The existing parameters SecurityHash will be renamded to IntegrityHash.

Proposed Solution

  • Feature specification is ready.
  • JADE-261 - Getting issue details... STATUS

Checksum File

Current Situation

  • A checksum file with the extension .md5 is created if 
    • the parameter CreateSecurityHashFile is set to trueand
    • the parameter CreateSecurityHash is set to true.
  • This feature is only available if 
    • the source protocol local is used or
    • a jump host is being used with the operation copytointernet.
  • If the parameter file_compress is used then the md5 value in the checksum file is calculated from the non-compressed input file. The checksum file itself is compressed.

Desired Behavior

  • If CreateSecurityHashFile=true the integrity hash of the target file is stored in a file on the target. 
    This file has the same name plus the integrity hash type (md5) as extension.
  • This feature works independent form the setting of the parameter CreateSecurityHash.
  • This feature shall be available independently from the protocol used for source and target systems.
  • If the parameter file_compress is used then the checksum file itself is not compressed.
  • If the parameters replacement and replacing are used then the name of the checksum file is created from the resulting name of the file on the target system.
  • The existing parameters SecurityHash will be renamded to IntegrityHash.

Proposed Solution

  • Feature specification is ready.
  • JADE-262 - Getting issue details... STATUS

 

 

  • No labels