Proposals for next features

Anchor
concurrent_transfer concurrent_transfer

Concurrent Transfer

Starting Situation

• YADE does not transfers multiple files in parallel.
• Currently one file is being processed by one channel at a time.

Desired Behavior

• Mutliple files should be transferred in parallel by use of different channels.

Proposed Solution

• Use multiple channels for file transfer
• See 

  • Jira
    server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-402

Anchor
large_file_handling large_file_handling

Large File Handling

Starting Situation

• JADE transfers multiple files in parallel.
• Currently YADE transfers one file is being processed by one channel at a time.

Desired Behavior

• Concerning the transfer of large files (> 1GB) the performance when using one channel for a file is too slow.
• Mutliple channels should be used.

Proposed Solution

• Using multiple channels would increase the overall speed of transfer
  • Use of multiple TCP streams to transfer data in parallel
• See also
  • http://monalisa.cern.ch/FDT/documentation_fdt.html implements the use of multiple channels

Anchor
resume_transfer resume_transfer

Resume File Transfer

Starting Situation

• If an error occurs during transfer of a file then this file has to be re-transferred completely.

Desired Behavior

• Automatically detect and resume the transfer after interruptions (e.g. connection is brokenlost).
  • Detect 
    • JADE YADE knows when a transfer was interrupted and would resume operation with the same parameters.
    • Detection is limited for a configurable duration after interrupted file transfers.
  • Resume
    • JADE YADE knows the chunks that have been transferred and is capable to continue writing chunks.
  • Fail-safe operation
    • JADE YADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file.

Proposed Solution

• Implementation with the JADE YADE Client.
• Use the history of file transfers to check for interrupted transfers.
• Keep track of chunks that have been transferred successfully in the file transfer history.Development of the functionality to detect and resume file transfers.

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-405

Anchor
prioritize_transfer prioritize_transfer

Priority of files to be transferred

Starting Situation

• If a group of files is to be transferred, e.g. files matched by a regular expression, then they are transferred in the order of sequence that is povided from that selection, which is somewhat arbitrary.

Desired Behaviour

• Individual files of a group of files should be prioritized within one transfer session.

Proposed Solution

• This feature could be based on the use of a list of files as input parameter for file transfer. 
  • By pre-processing (external command or internal rules) this list would be sorted before transfer.
  • Multiple file lists could be specified which would result in the fact that the files given in each list would be completely transferred before processing the next list.
  • The sequence in which lists are specified would cause JADE YADE to complete the transfer of each list before processing the next list, thus implementing prioritization by use of different lists.
  • Use of channels as specified in the Large File Handling feature should be focussed on prioritized files.
• Workaround for the current release: JADE YADE supports to use a list of files as input parameter. Transfers are then performed in the sequence of that list. During pre-processing that list could be created and sorted by priority.

Anchor
jms_interface jms_interface

JMS Interface

Starting Situation

• JADE YADE does not dispose of a JMS interface.

Desired Behaviour

• JADE YADE should implement a JMS interface to
  • publish messages to subscribers for a file transfer topic that would signal that a file has been transferred.
  • subscribe to file transfer topics that would cause JADE YADE to start a file transfer having received specific messages..
• Some use cases are required to elabore the requirements for this feature.

Proposed Solution

• Implementation with Apache Message Queue
• Possibly implemented for JobScheduler that would run the respective JADE YADE jobs based on events as the receipt of messages.
• Detailed specification required.
• As a prerequisite the YADE configuration is moved towards an XML format that allows later on to be assembled from XML fragments
  

  Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JADE-313

   
• Development of this feature is addressed with
  

  Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-127

   

Anchor
nfs_support nfs_support

NFS Support

Starting Situation 

• JADE YADE does not provide NFS support. 
• Therefore no Unix file systems using this protocol can be accessed directly by JADEYADE.
• Currently the respective file systems have to be mounted to a Unix system to make them accessible to JADEYADE.

Desired Behavior

• Support for NFS Version 2,3,4
• Have JADE YADE access any file system using NFS in the network.

Proposed Solution

• Possible implementation with libraries from Sun (http://java.net/projects/yanfs) or dCache (https://rb.dcache.org/r/). This is not yet decided on. 

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-404

Anchor
http_transfer http_transfer

HTTP, HTTPS as Data Provider

Starting Situation

• This feature is implemented with limits. Missing functionalities are:
  • proxy support
     
  • support for directory listings
• This feature is broken in the current JADE YADE release 
  • Due to the recently introduced mutli-threading this feature currently does not work.
  • It This will be fixed with the upcoming release Release 1.7.4321 (1.7.4)
    

Desired Behaviour

• Have proxy support and directory listings implemented.
• Have the HTTP, HTTPS data provider fixed to work as documented.

Proposed Solution

• Proxy Support
  • support for socks4 and socks5 proxies
    

    Jira
    server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-274

     
• Implementation based on Apache Commons VFS:
• support for directory listing, authentication.
• no support for write operations is provided.
  

  Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-287

   

Anchor
ldap_integration ldap_integration

LDAP Integration

Starting Situation

• No authentication or authorization is available for JADEYADE.
• JADE YADE is mainly used for batch file transfer, therefore hardly any indvidual users are involved that would have to be authenticated and authorized.

Desired Behavior

• JADE YADE should authenticate to an LDAP directory before performing transfers.
• Some use cases would be helpful to get a better idea of the scope of this feature, i.e. if the file transfer itself, its history GUI and/or JOC as the management interface for file transfers should authenticate.

Proposed Solution

• Authentication of JobScheduler components makes use of Apache Shiro. The same would apply to JADE YADE compoents.
• JADE YADE Client authentication with LDAP. 
• JADE YADE Background Service History GUI supports LDAP authentication.
  • This component will be available with Release 1.8.0
• JOC should support LDAP
  • This could be part of a major re-write of JOC that is envisaged for the next months, see JOC - Cockpit - Planning
  • We are currently in the process of collecting requirements for a new JOC Cockpit.

Anchor
publish_subscribe_model publish_subscribe_model

Publish-Subscribe Model

Starting Situation

• JADE YADE does not support to transfer files to multiple destinations, see Can JADE YADE use mutliple multiple destinations for file transfer?
• JADE YADE does not support the configuration management of routing rules for a publish-subscribe model

Desired Behavior

• JADE YADE should allow 1-to-many file transfers.
• Such file transfers should be based on a publish-subscribe model as proposed with the feature JMS Interface.
• JADE YADE should provide routing capabilities that would be managed centrally.

Proposed Solution

• Publish-subscribe models are often used with ESB systems, therefore no individual development of that model for JADE YADE is intended. Instead, JADE YADE should implement interfaces to popular ESB systems and use the models offered by ESBs, e.g.
  • Apache ServiceMix, Mule
•  JADE YADE should support an interface to routing components that are typically used for ESBs, e.g.
  • Apache Camel
• JADE YADE should not implement its own configuration management for routing rules but make use of configuration items of a central routing component.
• As a prerequisite the YADE configuration is moved towards an XML format that allows later on to be assembled from XML fragments.
  

  Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JADE-313

   

Anchor
monitoring_interface monitoring_interface

Monitoring Interface

Starting Situation

• JADE YADE supports notifications to System Monitors, e.g. Nagios, op5, Zabbix, if used as a JITL Job in JobScheduler.
• Additional System Monitors can easily be integrated should a command line interface be available.
• Other monitoring solutions with more sophisticated interfaces should be supported.

Desired Behavior

• File transfers effected by the JADE YADE Client should be subject to monitoring in the same way as the current JADE YADE JITL job.
• More Monitors such as IBM Tivoli, HP OVO, IBM WS Message, Broker should be supported.
• SNMP should be supported for interfacing with System Monitors, see Feature Proposal - JobScheduler SNMP Support

Proposed Solution

• Start from the feature JobScheduler Monitoring Interface
  • add monitoring capabilities for the file transfer history as reported by the JADE YADE Background Service
• Implement interfaces for popular Monitoring Systems
  • IBM Tivoli Monitoring: this should be feasible by a client command line tool and therefore be available with the JobScheduler Monitoring Interface.
  • HP OVO: to be clarified if a command line tool exists.
  • IBM WS Message Broker (MQ): to be clarified if a message queue interface were required or a command line tool were sufficient.
• SNMP Interface
  • SNMP could be supported via a separate process that integrates with the JobScheduler Monitoring interface.
    • the overall availability of jobs can be reported
    • in case of job errors respective messages are sent to an SNMP server.
  • SNMP is restricted to predefined messages. 
    • Though this is perfect for interfacing with all sorts of System Monitors the messages would not be too specific as the were defined in a standard MIB file
    • Such messages would include codes as e.g. for "ERROR-xxx File transfer failed". "ERROR-yyy Target host not found", however, they would not include specific information on the respective files, hosts etc.
  • Therefore, we agree that this is a desirable feature and at the same time we suggest to use the more versatile JobScheduler Monitoring Interface with System Monitors that would transmit more detailed information as it is required in case of errors in file transfers.
• For feature availability with the YADE JITL Job see
  

  Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JS-1299

Anchor
transfer_speed_restriction transfer_speed_restriction

Transfer Speed Restrictions

Starting situation

• JADE YADE uses all the available bandwith for a transfer and leaves it to the network administration to manage rules for bandwidth usage.

Desired Behavior

• JADE YADE should be configurable for use of bandwidth, e.g. for use in slow networks.

Proposed Solution

• Technical specification required

Anchor
file_integrity_checking file_integrity_checking

File Integrity Checking

Starting Situation

• JADE YADE has the following parameters around the integrity hash
  
  • CreateSecurityHash (default true)
  • CreateSecurityHashFile (default false)
  • CheckSecurityHash (default false)
  • SecurityHashType (default MD5)
• If CreateSecurityHash=true then only the integrity hash of the target file is calculated.
  • If CheckSecurityHash=false then CreateSecurityHashFile=true has no effect and the integrity hash file is not created.
    However, JADE YADE tries to transfer an integrity hash file which doesn't exist and subsequently an error is thrown.
• A check of the integrity hash is not implemented. In particular, the integrity hash of the source file has to be calculated too.
  • If CheckSecurityHash=true then JADE YADE tries to transfer a security hash file which doesn't necessarily exist and an error is thrown.
• If CreateSecurityHashFile=true then the security hash file will be created on the source but
  
  • ... it can only be created if the source has the "local" protocol
  • ... an error occurred if the source protocol is not "local".

Desired Behavior

• A integrity hash is created for source files and for target files (not for intermediate files on a jump host) to check if the transfer was complete and accurate.
• Integrity is checked by creating an integrity hash per source file and per target file and by comparing the checksum results.
• The parameter CreateSecurityHash will be removed as it is replaced by the parameter CheckSecurityHash.
• If CheckSecurityHash=true then the integrity hash of the source and the target file are calculated and will be checked after the transfer.
  • If the hashes are unequal then the transfer rolls back.
• The existing parameters SecurityHash will be renamded to IntegrityHash.

Proposed Solution

• Feature specification is ready.

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JADEYADE-261

Anchor
checksum_file checksum_file

Checksum File

Current Situation

• A checksum file with the extension .md5 is created if 
  • the parameter CreateSecurityHashFile is set to true and
  • the parameter CreateSecurityHash is set to true.
• A checksum file with the extension .md5 can be used from a source host if
  • • the parameter ReadSecurityHashFile is set to true and
    • the parameter CreateSecurityHash is set to true.

• This feature is only available if 
  • the source protocol local is used or
  • a jump host is being used with the operation copytointernet.
• If the parameter file_compress is used then the md5 value in the checksum file is calculated from the non-compressed input file. The checksum file itself is compressed.

Desired Behavior

• If CreateSecurityHashFile=true the integrity hash of the target file is stored in a file on the target. 
  This file has the same name plus the integrity hash type (md5) as extension.
• This feature works independent form the setting of the parameter CreateSecurityHash.
• This feature shall be available independently from the protocol used for source and target systems.
• If the parameter file_compress is used then the checksum file itself is not compressed.
• If the parameters replacement and replacing are used then the name of the checksum file is created from the resulting name of the file on the target system.
• The existing parameters SecurityHash will be renamded to IntegrityHash.

Proposed Solution

• Feature specification is ready.

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JADEYADE-262