Product Knowledge Base

Space shortcuts

Page tree

Scenario

In this example we want to poll for an incoming file on the Server A, once file(s) matching a reguler expression are found on Server A then JobScheduler should transfer the file(s) from Server A to Server B.
After successful transfer of files to Server B then JobScheduler should execute a script for post-processing on Server B via SSH.

Solution

  • For such a file transfer and processing workflow JobScheduler has built-in JITL jobs, namely the YADE job and SSH job.
  • The YADE JITL Jobs be configured to transfer data from Server-to-Server without touchdown, more information about Server-to-Server transfer can be found here: Example for YADE Server-to-Server File Transfer.
  • YADE has a built-in polling feature that can be configured by using polling parameters, i.e.
    • poll_interval: frequency of polling in seconds and
    • poll_timeout: maximum duration in seconds for polling.
  • The job chain can be configured by an order to run at a predefined time, to repeat execution or to be started manually.

Job Chain: FileTransferAndProcessing.job_chain.xml

The job chain has two nodes:

  1. jade_s2s_file_transfer: for server to server data transfer and
  2. ssh_start_proccessing: starting the processing script using ssh
     
<?xml version="1.0" encoding="ISO-8859-1"?>
<job_chain  orders_recoverable="yes" visible="yes" title="Tranasfer Data from ServerA to ServerB and start Processing on ServerB vis SSH">

    <job_chain_node  state="jade_s2s_file_transfer" job="jade_s2s_file_transfer" error_state="error" next_state="ssh_start_proccessing" on_error="suspend"/>
    <job_chain_node  state="ssh_start_proccessing" job="ssh_start_proccessing" next_state="success" error_state="error" on_error="suspend"/>
    <job_chain_node  state="success"/>
    <job_chain_node  state="error"/>

</job_chain>
Step 1: Job jade_s2s_file_transfer
  • This job is a YADE JITL job that is configured to poll a source host (via FTP) for files matching file_spec=^TEST*\.log$ for every 30 seconds during a maximum of 600 seconds.
  • The matching file(s) will be copied to the target host (SFTP). More information about YADE JITL job configuration can be found here: JITL Jobs by topic.
Job jade_s2s_file_transfer 
<?xml version="1.0" encoding="ISO-8859-1"?>
<job  title="S2S transfer with polling (API Job for JobScheduler Advanced Data Exchange)" order="yes">
    <description >
        <include  file="jobs/jadeJob.xml"/>
    </description>

    <params >
        <param  name="file_spec" value="\.log$"/>
        <param  name="operation" value="copy"/>
        <param  name="verbose" value="9"/>
        <param  name="poll_interval" value="30"/>
        <param  name="poll_minfiles" value="1"/>
        <param  name="poll_timeout" value="600"/>
        <param  name="source_protocol" value="ftp"/>
        <param  name="source_host" value="mp.sos"/>
        <param  name="source_user" value="test"/>
        <param  name="source_password" value="super_secret"/>
        <param  name="source_dir" value="/outbound"/>
        <param  name="target_protocol" value="sftp"/>
        <param  name="target_host" value="homer.sos"/>
        <param  name="target_user" value="test"/>
        <param  name="target_password" value="super_secret"/>
        <param  name="target_ssh_auth_method" value="password"/>
        <param  name="target_dir" value="/home/test/inbound"/>
    </params>

    <script  language="java" java_class="sos.scheduler.jade.JadeJob"/>

    <run_time />
</job>
Step 2: Job ssh_start_proccessing
  • This job is a JITL job that is configured to execute a command on a remote host using SSH. The SSH JITL job can be configured to connect using a public/private key pair or with user/password authentication.
  • More information about SSH JITL job configuration can be found here: SSH_JITL_JOB_PARAMETERS. 
Job ssh_start_proccessing 
<?xml version="1.0" encoding="ISO-8859-1"?>
<job  title="Launch commands or executable files by SSH" order="yes">
    <description >
        <include  file="jobs/SOSSSHJob2JSAdapter.xml"/>
    </description>

    <params >
        <param  name="host" value="homer.sos"/>
        <param  name="port" value="22"/>
        <param  name="user" value="test"/>
        <param  name="auth_method" value="password"/>
        <param  name="command" value="uname -an"/>
        <param  name="password" value="super_secret"/>
    </params>

    <script  language="java" java_class="sos.scheduler.job.SOSSSHJob2JSAdapter"/>

    <run_time />
</job>

Related Downloads

See also