In this example we want to poll for an incoming file on the Server A, once file(s) matching a reguler expression are found on Server A then JobScheduler should transfer the file(s) from Server A to Server B.
After successful transfer of files to Server B then JobScheduler should execute a script for post-processing on Server B via SSH.
- For such a file transfer and processing workflow JobScheduler has built-in JITL jobs, namely the YADE job and SSH job.
- The YADE JITL Jobs be configured to transfer data from Server-to-Server without touchdown, more information about Server-to-Server transfer can be found here: Example for YADE Server-to-Server File Transfer.
- YADE has a built-in polling feature that can be configured by using polling parameters, i.e.
- poll_interval: frequency of polling in seconds and
- poll_timeout: maximum duration in seconds for polling.
- The job chain can be configured by an order to run at a predefined time, to repeat execution or to be started manually.
Job Chain: FileTransferAndProcessing.job_chain.xml
The job chain has two nodes:
- jade_s2s_file_transfer: for server to server data transfer and
- ssh_start_proccessing: starting the processing script using ssh
Step 1: Job jade_s2s_file_transfer
- This job is a YADE JITL job that is configured to poll a source host (via FTP) for files matching file_spec=^TEST*\.log$ for every 30 seconds during a maximum of 600 seconds.
- The matching file(s) will be copied to the target host (SFTP). More information about YADE JITL job configuration can be found here: JITL Jobs by topic.
Step 2: Job ssh_start_proccessing
- This job is a JITL job that is configured to execute a command on a remote host using SSH. The SSH JITL job can be configured to connect using a public/private key pair or with user/password authentication.
- More information about SSH JITL job configuration can be found here: SSH_JITL_JOB_PARAMETERS.
- JobScheduler job_chain and jobs: download example
- YADE Implementation Architecture
- How to transfer files with YADE from Server A to Server B using polling and post-processing
- Example for YADE Server-to-Server File Transfer
- Example for YADE Server-to-Server File Transfer with polling