Product Knowledge Base

Space shortcuts

Page tree

Proposals for next features

Concurrent Transfer

Starting Situation

  • YADE does not transfers multiple files in parallel.
  • Currently one file is being processed by one channel at a time.

Desired Behavior

  • Mutliple files should be transferred in parallel by use of different channels.

Proposed Solution

  • Use multiple channels for file transfer
  • See 

Large File Handling

Starting Situation

  • YADE transfers one file by one channel at a time.

Desired Behavior

  • Concerning the transfer of large files (> 1GB) the performance when using one channel for a file is too slow.
  • Mutliple channels should be used.

Proposed Solution

Resume File Transfer

Starting Situation

  • If an error occurs during transfer of a file then this file has to be re-transferred completely.

Desired Behavior

  • Automatically detect and resume the transfer after interruptions (e.g. connection is lost).
    • Detect 
      • YADE knows when a transfer was interrupted and would resume operation with the same parameters.
      • Detection is limited for a configurable duration after interrupted file transfers.
    • Resume
      • YADE knows the chunks that have been transferred and is capable to continue writing chunks.
    • Fail-safe operation
      • YADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file.

Proposed Solution

  • Implementation with the YADE Client.
  • Keep track of chunks that have been transferred successfully in the file transfer history.
  • YADE-405 - Getting issue details... STATUS

Priority of files to be transferred

Starting Situation

  • If a group of files is to be transferred, e.g. files matched by a regular expression, then they are transferred in the order of sequence that is povided from that selection, which is somewhat arbitrary.

Desired Behaviour

  • Individual files of a group of files should be prioritized within one transfer session.

Proposed Solution

  • This feature could be based on the use of a list of files as input parameter for file transfer. 
    • By pre-processing (external command or internal rules) this list would be sorted before transfer.
    • Multiple file lists could be specified which would result in the fact that the files given in each list would be completely transferred before processing the next list.
    • The sequence in which lists are specified would cause YADE to complete the transfer of each list before processing the next list, thus implementing prioritization by use of different lists.
    • Use of channels as specified in the Large File Handling feature should be focussed on prioritized files.
  • Workaround for the current release: YADE supports to use a list of files as input parameter. Transfers are then performed in the sequence of that list. During pre-processing that list could be created and sorted by priority.

JMS Interface

Starting Situation

  • YADE does not dispose of a JMS interface.

Desired Behaviour

  • YADE should implement a JMS interface to
    • publish messages to subscribers for a file transfer topic that would signal that a file has been transferred.
    • subscribe to file transfer topics that would cause YADE to start a file transfer having received specific messages..
  • Some use cases are required to elabore the requirements for this feature.

Proposed Solution

  • Implementation with Apache Message Queue
  • Possibly implemented for JobScheduler that would run the respective YADE jobs based on events as the receipt of messages.
  • Detailed specification required.
  • As a prerequisite the YADE configuration is moved towards an XML format that allows later on to be assembled from XML fragments
    JADE-313 - Getting issue details... STATUS  
  • Development of this feature is addressed with
    YADE-127 - Getting issue details... STATUS  

NFS Support

Starting Situation 

  • YADE does not provide NFS support. 
  • Therefore no Unix file systems using this protocol can be accessed directly by YADE.
  • Currently the respective file systems have to be mounted to a Unix system to make them accessible to YADE.

Desired Behavior

  • Support for NFS Version 2,3,4
  • Have YADE access any file system using NFS in the network.

Proposed Solution

HTTP, HTTPS as Data Provider

Starting Situation

  • This feature is implemented with limits. Missing functionalities are:
    • proxy support
       
    • support for directory listings
  • This feature is broken in the current YADE release 
    • Due to the recently introduced mutli-threading this feature currently does not work.
    • This will be fixed with release Release 1.7.4321 (1.7.4)

Desired Behaviour

  • Have proxy support and directory listings implemented.
  • Have the HTTP, HTTPS data provider fixed to work as documented.

Proposed Solution

  • Proxy Support
    • support for socks4 and socks5 proxies
      YADE-274 - Getting issue details... STATUS  
  • Implementation based on Apache Commons VFS:
    • support for directory listing, authentication.
    • support for write operations is provided.
      YADE-287 - Getting issue details... STATUS  

LDAP Integration

Starting Situation

  • No authentication or authorization is available for YADE.
  • YADE is mainly used for batch file transfer, therefore hardly any indvidual users are involved that would have to be authenticated and authorized.

Desired Behavior

  • YADE should authenticate to an LDAP directory before performing transfers.
  • Some use cases would be helpful to get a better idea of the scope of this feature, i.e. if the file transfer itself, its history GUI and/or JOC as the management interface for file transfers should authenticate.

Proposed Solution

  • Authentication of JobScheduler components makes use of Apache Shiro. The same would apply to YADE compoents.
  • YADE Client authentication with LDAP. 
  • YADE Background Service History GUI supports LDAP authentication.
  • JOC should support LDAP
    • This could be part of a major re-write of JOC that is envisaged for the next months, see JOC - Cockpit - Planning
    • We are currently in the process of collecting requirements for a new JOC Cockpit.

Publish-Subscribe Model

Starting Situation

Desired Behavior

  • YADE should allow 1-to-many file transfers.
  • Such file transfers should be based on a publish-subscribe model as proposed with the feature JMS Interface.
  • YADE should provide routing capabilities that would be managed centrally.

Proposed Solution

  • Publish-subscribe models are often used with ESB systems, therefore no individual development of that model for YADE is intended. Instead, YADE should implement interfaces to popular ESB systems and use the models offered by ESBs, e.g.
    • Apache ServiceMix, Mule
  • YADE should support an interface to routing components that are typically used for ESBs, e.g.
    • Apache Camel
  • YADE should not implement its own configuration management for routing rules but make use of configuration items of a central routing component.
  • As a prerequisite the YADE configuration is moved towards an XML format that allows later on to be assembled from XML fragments.
    JADE-313 - Getting issue details... STATUS  

Monitoring Interface

Starting Situation

  • YADE supports notifications to System Monitors, e.g. Nagios, op5, Zabbix, if used as a JITL Job in JobScheduler.
  • Additional System Monitors can easily be integrated should a command line interface be available.
  • Other monitoring solutions with more sophisticated interfaces should be supported.

Desired Behavior

  • File transfers effected by the YADE Client should be subject to monitoring in the same way as the current YADE JITL job.
  • More Monitors such as IBM Tivoli, HP OVO, IBM WS Message, Broker should be supported.
  • SNMP should be supported for interfacing with System Monitors, see Feature Proposal - JobScheduler SNMP Support

Proposed Solution

  • Start from the feature JobScheduler Monitoring Interface
    • add monitoring capabilities for the file transfer history as reported by the YADE Background Service
  • Implement interfaces for popular Monitoring Systems
    • IBM Tivoli Monitoring: this should be feasible by a client command line tool and therefore be available with the JobScheduler Monitoring Interface.
    • HP OVO: to be clarified if a command line tool exists.
    • IBM WS Message Broker (MQ): to be clarified if a message queue interface were required or a command line tool were sufficient.
  • SNMP Interface
    • SNMP could be supported via a separate process that integrates with the JobScheduler Monitoring interface.
      • the overall availability of jobs can be reported
      • in case of job errors respective messages are sent to an SNMP server.
    • SNMP is restricted to predefined messages. 
      • Though this is perfect for interfacing with all sorts of System Monitors the messages would not be too specific as the were defined in a standard MIB file
      • Such messages would include codes as e.g. for "ERROR-xxx File transfer failed". "ERROR-yyy Target host not found", however, they would not include specific information on the respective files, hosts etc.
    • Therefore, we agree that this is a desirable feature and at the same time we suggest to use the more versatile JobScheduler Monitoring Interface with System Monitors that would transmit more detailed information as it is required in case of errors in file transfers.
  • For feature availability with the YADE JITL Job see
    JS-1299 - Getting issue details... STATUS

Transfer Speed Restrictions

Starting situation

  • YADE uses all the available bandwith for a transfer and leaves it to the network administration to manage rules for bandwidth usage.

Desired Behavior

  • YADE should be configurable for use of bandwidth, e.g. for use in slow networks.

Proposed Solution

  • Technical specification required

File Integrity Checking

Starting Situation

  • YADE has the following parameters around the integrity hash
    • CreateSecurityHash (default true)
    • CreateSecurityHashFile (default false)
    • CheckSecurityHash (default false)
    • SecurityHashType (default MD5)
  • If CreateSecurityHash=true then only the integrity hash of the target file is calculated.
    • If CheckSecurityHash=false then CreateSecurityHashFile=true has no effect and the integrity hash file is not created.
      However, YADE tries to transfer an integrity hash file which doesn't exist and subsequently an error is thrown.
  • A check of the integrity hash is not implemented. In particular, the integrity hash of the source file has to be calculated too.
    • If CheckSecurityHash=true then YADE tries to transfer a security hash file which doesn't necessarily exist and an error is thrown.
  • If CreateSecurityHashFile=true then the security hash file will be created on the source but
    • ... it can only be created if the source has the "local" protocol
    • ... an error occurred if the source protocol is not "local".

Desired Behavior

  • A integrity hash is created for source files and for target files (not for intermediate files on a jump host) to check if the transfer was complete and accurate.
  • Integrity is checked by creating an integrity hash per source file and per target file and by comparing the checksum results.
  • The parameter CreateSecurityHash will be removed as it is replaced by the parameter CheckSecurityHash.
  • If CheckSecurityHash=true then the integrity hash of the source and the target file are calculated and will be checked after the transfer.
    • If the hashes are unequal then the transfer rolls back.
  • The existing parameters SecurityHash will be renamded to IntegrityHash.

Proposed Solution

  • Feature specification is ready.
  • YADE-261 - Getting issue details... STATUS

Checksum File

Current Situation

  • A checksum file with the extension .md5 is created if 
    • the parameter CreateSecurityHashFile is set to true and
    • the parameter CreateSecurityHash is set to true.
  • A checksum file with the extension .md5 can be used from a source host if
      • the parameter ReadSecurityHashFile is set to true and
      • the parameter CreateSecurityHash is set to true.
  • This feature is only available if 
    • the source protocol local is used or
    • a jump host is being used with the operation copytointernet.
  • If the parameter file_compress is used then the md5 value in the checksum file is calculated from the non-compressed input file. The checksum file itself is compressed.

Desired Behavior

  • If CreateSecurityHashFile=true the integrity hash of the target file is stored in a file on the target. 
    This file has the same name plus the integrity hash type (md5) as extension.
  • This feature works independent form the setting of the parameter CreateSecurityHash.
  • This feature shall be available independently from the protocol used for source and target systems.
  • If the parameter file_compress is used then the checksum file itself is not compressed.
  • If the parameters replacement and replacing are used then the name of the checksum file is created from the resulting name of the file on the target system.
  • The existing parameters SecurityHash will be renamded to IntegrityHash.

Proposed Solution

  • Feature specification is ready.
  • YADE-262 - Getting issue details... STATUS