How to transfer data from ServerA to ServerB using JADE job with polling and start ssh processing job on ServerB

Scenario

In this example user want to poll for an incoming file on the ServerA, once file matching with the reguler expression is found on ServerA, JobScheduler should transfer the file(s) from ServerA to ServerB.
After successfully transfer of file on ServerB, JobScheduler should start an processing script on ServerB via SSH.

Warning: : syntax error in line 1 near '\'

Solution

For such a file transfer and processing workflow, JobScheduler has built in JITL jobs, namely JADE job and SSH job.
The JADE JITL jobs can be configured to transfer data from Server-to-Server without touch down, more information about Server-to-Server transfer can be found here Server to Server Transfer.
The JADE has inbuilt polling feature which can be configured using polling parameter of JADE i.e. poll_interval : frequency of polling in seconds and poll_timeout : maximum time in seconds for polling. The job_chain can be configured by an order to run at predefined time, repeat execution or can be started manually.

Job Chain : FileTransferAndProcessing.job_chain.xml

The job chain FileTransferAndProcessing has two nodes:

jade_s2s_file_transfer : for server to server data transfer and
ssh_start_proccessing : starting the processing script using ssh

<?xml version="1.0" encoding="ISO-8859-1"?>


<job_chain  orders_recoverable="yes" visible="yes" title="Tranasfer Data from ServerA to ServerB and start Processing on ServerB vis SSH" max_orders="1" name="DataTransferProcessing">
    <job_chain_node  state="jade_s2s_file_transfer" job="jade_s2s_file_transfer" error_state="error" next_state="ssh_start_proccessing" on_error="suspend"/>

    <job_chain_node  state="ssh_start_proccessing" job="ssh_start_proccessing" next_state="success" error_state="error" on_error="suspend"/>

    <job_chain_node  state="success"/>

    <job_chain_node  state="error"/>
</job_chain>

Step 1: jade_s2s_file_transfer.job.xml

This job is an JADE JITL job which is configured to poll source server (ftp) for file matching with file_speh1. ^TEST*\.log$ for every 30 seconds for maximum of 600 seconds.
All the matching file(s) during polling will be copy(ied) to the target server (sftp). For more information about JADE JITL job can be found here JADE_JITL_Parameters.



<?xml version="1.0" encoding="ISO-8859-1"?>


<job  title="S2S transfer with polling (API Job for JobScheduler Advanced Data Exchange)" order="yes" stop_on_error="no" name="jade_s2s_file_transfer">
    <description >
        <include  file="jobs/jadeJob.xml"/>
    </description>

    <params >
        <param  name="file_spec" value="\.log$"/>

        <param  name="operation" value="copy"/>

        <param  name="verbose" value="9"/>

        
        <param  name="poll_interval" value="30"/>

        <param  name="poll_minfiles" value="1"/>

        <param  name="poll_timeout" value="600"/>

        
        

        <param  name="source_protocol" value="ftp"/>

        <param  name="source_host" value="mp.sos"/>

        <param  name="source_user" value="test"/>

        <param  name="source_password" value="super_secret"/>

        <param  name="source_dir" value="/outbound"/>

        

        <param  name="target_protocol" value="sftp"/>

        <param  name="target_host" value="homer.sos"/>

        <param  name="target_user" value="test"/>

        <param  name="target_password" value="super_secret"/>

        <param  name="target_ssh_auth_method" value="password"/>

        <param  name="target_dir" value="/home/test/inbound"/>
    </params>

    <script  language="java" java_class="sos.scheduler.jade.JadeJob"/>

    <run_time />
</job>

Step 2: ssh_start_proccessing.job.xml

This job is an JITL job which is configured to execute an command on remote server using ssh. The SSH JITL job can be configured to connect using Private/public key pair, or with userid/password.
More information about SSH JITL job can be found here SSH_JITL_JOB_PARAMETERS.

<?xml version="1.0" encoding="ISO-8859-1"?>


<job  title="Launch commands or executable files by SSH" order="yes" name="ssh_start_proccessing">
    <description >
        <include  file="jobs/SOSSSHJob2JSAdapter.xml"/>
    </description>

    <params >
        <param  name="host" value="homer.sos"/>

        <param  name="port" value="22"/>

        <param  name="user" value="test"/>

        <param  name="auth_method" value="password"/>

        <param  name="command" value="uname -an"/>

        <param  name="password" value="super_secret"/>
    </params>

    <script  language="java" java_class="sos.scheduler.job.SOSSSHJob2JSAdapter"/>

    <run_time />
</job>

Also see

Download example

JobScheduler job_chain/jobs : download example

<mscgen>
// Remote configuration
msc {

 width"800";

config_server labeh1. "Configuration Server JobScheduler",workload label"Workload JobScheduler";

config_server h1. workload label "First Request HTTP using JETTY Port)";

  config_server box config_server[labeh1. "Register Client", textbgcolour"#7f7fff",  textcolouh1. "#FFFFFF"];
  config_server < workload  [labelh1. "Request every 15 minutes (HTTP using JETTY Port)"];
  config_server> workload  [labelh1. "Sending configuration changes (HTTP using JETTY Port)"];
    ---  [label "loop every 15 minutes", ID="*"];

}
</mscgen>

Space shortcuts

Page tree