Page History

...

Organising secure file transfer with the JobScheduler

Starting point: We have an explicit need to invoke an FTP get file operation when a file becomes available on a remote server, assuming that

...

Question: Is there any way to monitor a remote file directory or folder on the job schedule so that a notification is received on the main JobScheduler to start the FTP get file program or script?

Proposed Solution 1: Signalling

Should the remote host be responsible to signal the existence of a file ready for transfer, then concerning the remote host don't be afraid of the JobScheduler API, it might be as simple as this:

...

This command is most likely available for the remote host and it's simple: by means of a TCP stack you could send an XML command that tells the JobScheduler what should be done, e.g. to start a job. Major parts of the API are available as XML commands that could be sent via TCP or UDP to the JobScheduler. You do not have to use telnet as the ultimate TCP solution, it's just an example that works at the command line.

Use the JobScheduler API without installation

We deliver Java classes that could be used without a JobScheduler being installed, e.g. on the command line like this:

...

Java is available for most operating systems, therefore the remote host could use this command line to send any XML command to the JobScheduler without he need to have a JobScheduler being installed on the remote host. A standard JRE and the above .jar archives would be sufficient for the remote host.

Use an alternative TCP/UDP stack

• You could use any programming or scripting language that supports TCP or UDP to send XML commands.
• The above use cases do not assume that you have to maintain the resp. scripts or programs on the remote host as standard operating system utilities could be used for this. 
• We recommend to use UDP in the above solution as this protocol is non-blocking, i.e. the remote host will not suffer (and have to recover) from errors due to network downtimes. Therefore use of signalling should be optional in the communication between remote host and JobScheduler to speed up processing but is no replacement for polling.

Proposed solution 2: Polling

We recommend this solution as it is save and easy to implement to check if a file exists on a remote host:

Check via FTP

See the standard jobs in the chapter for FTP processing at JITL - JobScheduler Integrated Template Library and the job documentation in JobScheduler FTP Receive.
The existing code could be adapted to return distinct exit codes that signal if files are available or absent in a remote directory.

Check via SSH

This solution requires a shell script or operating system command on the remote host that is launched via SSH by a job in order to check if a file is ready to be transferred. The script or command could be implemented to return a distinct exit code which signals that files are ready for transfer.

...

Starting successor jobs based on exit codes is not restricted to SSH or to the use of <process file=""/>, you could configure any job to start arbitrary successor jobs and orders (from JobScheduler release 1.2.7 on).

Concurrency Issues

A simple way to avoid concurrency would be to have the client put files with a name as filename~ and rename them to filename. This would make the files appear atomically.

Alternative solutions

More sophisticated solutions are required in the following cases:

...