Page History

Organising secure file transfer with the JobScheduler

...

Should the remote host be responsible to signal the existence of a file ready for transfer, then concerning the remote host don't be afraid of the JobScheduler API, it might be as simple as this:

 {{telnet <i>[scheduler_host]</i><i>[scheduler_port]</i>}} 
 {{<start_job job="ftp_get_files"></params></start_job>}} 

This command is most likely available for the remote host and its simple: by means of a TCP stack you could send an XML command that tells the JobScheduler what should be done, e.g. to start a job. Major parts of the API are available as XML commands that could be sent via TCP or UDP to the JobScheduler. You do not have to use {{telnet}} as the ultimate TCP solution, it's just an example that works at the command line.

...

we deliver Java classes that could be used without a JobScheduler being installed, e.g. on the command line like this:

 {{java  -cp ./sos.scheduler.jar:./sos.util.jar sos.scheduler.SOSSchedulerCommand <i>[scheduler_host] [scheduler_port] [xml command]</i>}} 

Java is available for most operating systems, therefore the remote host could use this command line to send any XML command to the JobScheduler without he need to have a JobScheduler being installed on the remote host. A standard JRE 1.6++ and the above .jar archives would be sufficient for the remote host.

...

See the standard jobs in the chapter for FTP processing at
JobScheduler Standard Jobs and the job documentation in JobScheduler FTP Receive.
The existing code could be adapted to return distinct exit codes that signal if files are available or absent in a remote directory.

...

A (pseudo) configuration for the job could be:

 <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/job.xml job ] <job >
   <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/process.xml process] <process file="ssh" param="-l [user]@[host] [script_or_command]">     
     <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/delay<delay_after_error.xml delay_after_error] error_count="1" delay="60"/>    
     <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/delay_after_error.xml delay<delay_after_error] error_count="10" delay="600"/>    
     <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/delay<delay_after_error.xml delay_after_error] error_count="50" delay="stop"/>    
          
     <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/commands.xml commands] <commands on_exit_code="success">
       <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/start_job.xml start_job]<start_job job="ftp_get_files">
          <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/params.xml params]/<params/>
       </start_job>
     </commands>
      
      
     <[http://www.sos-berlin.com/doc/en/scheduler.doc/xml/commands.xml commands]<commands on_exit_code="1"/> <!-- other exit codes are automatically handled as  errors, otherwise you could start individual jobs to handle errors ->
   
   </process>
 </job>

...

A simple way to avoid concurrency would be to have the client put files with a name as filename~ and rename them to filename. This would make the files appear atomically.

...

Alternative solutions

More sophisticated solutions are needed in the following cases:

• What should happen if one input file cannot be fetched for a longer period and subsequent files are provided by the remote host?
• How should multiple input files be handled that are to be processed in a single transaction by the client, e.g. for dbms import?
  We implemented commercial solutions for these issues based on traffic light rules.
  If this should be of any interest to you then please contact us.