How to organise secure file transfer with the JobScheduler

Organizing secure file transfer with the Job Scheduler

We have an explicit need to invoke an ftp get file operation when a file becomes available on a remote server,
assuming that:

The remote server operating system is not supported by the Job Scheduler.
We don't wish to rely on a program that uses the Job Scheduler API package running on the remote node to send us a notification (we can't maintain programs running on this remote node).
Is there any way to monitor a remote file directory or folder on the job schedule so that a notification is received on the main job scheduler to start the ftp get file program or script?
1. Proposed Solution: Signalling

Should the remote host be responsible to signal the existence of a file ready for transfer, then concerning the remote host
<ul clash1. "sos-list"><li class"sos-list-item">don't be afraid of the Job Scheduler API, it might be as simple as this:
telnet scheduler_hostscheduler_port
<start_job job="ftp_get_files"></params></start_job>

This command is most likely available for the remote host and its simple: by means of a TCP stack you could send an XML command that tells the Job Scheduler what should be done, e.g. to start a job. Major parts of the API are available as XML commands that could be sent via TCP or UDP to the Job Scheduler. You do not have to use{{telnet}} as the ultimate TCP solution, it's just a sample that works at the command line.
</li><li clash1. "sos-list-item">use the Job Scheduler API without installation

we deliver Java classes that could be used without a Job Scheduler being installed, e.g. on the command
line like this:
{{java -cp ./sos.scheduler.jar:./sos.util.jar sos.scheduler.SOSSchedulerCommand
scheduler_hostscheduler_portxml command}}

Java is available for most operating systems, therefore the remote host could use this command line to send any XML command to the Job Scheduler without he need to have a Job Scheduler being installed on the remote host. A standard JRE 1.6++ and the above .jar archives would be sufficient for the remote host.

use an alternative TCP/UDP stack

You could use any programming/scripting language that supports TCP or UDP to send XML commands.
The above cases do not assume that you have to maintain the resp. scripts/programs on the remote host as standard operating system utilities could be sed for this. We recommend to use UDP in the above solutions as this protocol is non-blocking, i.e. the remote host will not suffer (and have to recover) rom errors due to network downtimes. Therefore use of signalling should be optional in the communication between remote host and Job Scheduler to speed up processing but is no replacement for polling.

2. Proposed solution: polling

we recommend this solution as it is save and easy to implement to check if a file exists on a remote host:

Check via FTP

See the standard jobs in the chapter for FTP processing at
Job Scheduler Standard Jobs and the job documentation in Job Scheduler FTP Receive
The existing code could be adapted to return distinct exit codes that signal if files are available or absent in a remote directory.

Check via ssh

This solution requires a shell script or operating system command on the remote host that is launched via ssh by a job in order to check if a file is ready to be transferred. The script/command could be implemented to return a distinct exit code which signals that files are ready for transfer.

A (pseudo) configuration for the job could be:

 <job>
   <process file="ssh" param="-l [user]@[host] [script_or_command]">     
     <delay_after_error error_count="1" delay="60"/>    
     <delay_after_error error_count="10" delay="600"/>    
     <delay_after_error error_count="50" delay="stop"/>    
          
     <commands on_exit_code="success">
       <start_job job="ftp_get_files">
          <params/>
       </start_job>
     </commands>
      
      
     <commands on_exit_code="1"/> <!-- other exit codes are automatically handled as  errors, otherwise you could start individual jobs to handle errors ->
   
   </process>
 </job>

Starting successor jobs based on exit codes is not restricted to ssh or the use of <process file=""/>, you could configure any job to start arbitrary successor jobs and orders (from Job Scheduler release 1.2.7 on).
</li></ul></li><li class"sos-list-item">Concurrency Issues

A simple way to avoid concurrency would be to have the client put files with a name as filename~ and rename them to filename. This would make the files appear atomically.

Space shortcuts

Page tree

Organizing secure file transfer with the Job Scheduler

1. Proposed Solution: Signalling

use an alternative TCP/UDP stack

2. Proposed solution: polling

Check via FTP

Check via ssh

more solutions