How to define and reference job chain node parameters

Introduction

JobScheduler can reference job chain node parameters - i.e. parameters that are valid for a specific node in a job chain.

Job chain node parameters can be defined for orders and for the job chain:

• Job chain node parameters for orders are defined in a similar manner to conventional order parameters - but using a special syntax.
• Job chain node parameters for the job chain itself are defined in a job chain parameter file. The jobs in the job chain use a special monitor to read the job chain parameter file.

Job chain node parameters for both orders and for the job chain can reference other (global) parameters.

Usage Notes

In contrast with job parameters, job chain node parameters are independent of the the job specified for a job chain. Their use provides a greater degree of flexibility in job chain configuration than can be achieved with job or order parameters.

• Order parameters are - logically - tied to an order and have to be redefined each time an order is created.
• Similarly, job parameters are tied to a job. This means that if job parameters are used for a job that is to be used either at multiple steps in a job chain or in different job chains, multiple instances of that job will have do be configured if different parameters are to be set each time the job is to be run, even though it is the same job implementation that is being used each time.

Usage example

As an illustration of the flexibility brought with job chain node parameters, consider the following simple example:

For a job chain made up of three nodes:

1. transfer a file by FTP to host a,
2. transfer a file by FTP to host b,
3. delete file.

the use of job chain node parameters would allow the same job to be used for both nodes 1 and 2 but with different parameters each time.

Article Scope

Two example job chains are described in this article:

• The first uses job chain node parameters defined for an order.
• The second example uses job chain node parameters defined for the job chain itself and also demonstrates referencing of global parameters.

Example showing job chain node parameters defined for an order

This example shows how job chain node parameters can be configured in an order and assigned to specific job chain nodes.

In addition, the example shows how these parameters can be called in shell jobs and used together with 'standard' job and order parameters.

Example Download

You can download the example:

• node_parameters_from_order.zip.

Syntax for specifying job chain node parameters in orders

• The syntax for order parameters that are specific to a job chain node is <node>/<name>.
  (This is in contrast to the 'standard' order parameter syntax of <name>.)

Note on usage

• A job chain node parameter will be overwritten by an order node parameter with the same name.
• Order node parameters can be used if job chain node parameters have not been defined and a job chain configuration file has not been created.

Example Job Chain configuration

The following screenshot shows the example job chain with two nodes and the same job specified for each node:

The next screen shot shows the parameters defined for the order.

The first two entries in the parameter tab form show the values for the orderNodeParameter defined for the each of the nodes in the job chain:

• at the node 100 the orderNodeParam will have the value 100
• at node 200 the orderNodeParam will have the value 200 together with the values of two other parameters:
  • jobParam -  a job parameter specified for the job1 and
  • orderParam - an order parameter which is specified in the next line of the form 

The third entry in the form shows the 'standard' order parameter orderParam.

The next screenshot shows the jobParam parameter which retains its value every time the job is run.

The example has a very simple shell script that gives out the value of the orderNodeParam job chain node parameter.

JobScheduler exposes job chain node parameters to shell scripts as environment variables. As can be seen in the screenshot below, the parameter name is prefixed with the default JobScheduler environment variable prefix SCHEDULER_PARAM_.

Note that on Unix systems the required syntax would be $SCHEDULER_PARAM_ORDERNODEPARAM.

A pre-/post-processing configuration monitor with a Java class has to be specified for the job before the jobParam and orderParam parameters can be used in the job chain node parameters in the shell script. The configuration monitor is shown in the next screenshot:

As already described above, the values of the jobParam and orderParam parameters are substituted into the orderNodeParameter at the 200 node of the job chain.

Running the example

• Open the JobScheduler Operating Center, JOC, in your browser using http://scheduler_host:scheduler_port
• Open the JOB CHAINS view.
• Navigate to the folder node_parameters_from_order 
• Click Expand Details
• Start the order1 using the Order Menu / Start order now

• Opening the log file for the order will show that at step 100 the job1 shell script will give out the value for the orderNodeParam defined in the order:

  [stdout] job1: Value of orderNodeParam = 100

• At step 200 the shell script will give out the value for the orderNodeParam defined in the order (200) together with the values set for the orderParam and jobParam:

  [stdout] job1: Value of orderNodeParam = 200 Parameter-value-from-job - Parameter-value-from-order

Use with API Jobs

In releases 1.8.4, 1.9.8 and 1.10.2 the above example works with shell jobs and not with API jobs - which includes many of the JITL jobs.

An issue has been raised - for more information see JS-1573 - Getting issue details... STATUS

Workaround for API & JITL Jobs in release 1.8.4, 1.9.8, 1.10.2

To activate the order node function please add the configuration monitor to the pre-processing element of the job at the job node.

 Example for assigning a configuration monitor

  <job  stop_on_error="no" order="yes" name="job1">
     ....
    <monitor  name="configuration_monitor" ordering="0">
        <script  language="java" java_class_path="" java_class="sos.scheduler.managed.configuration.ConfigurationOrderMonitor"/>
    </monitor>
  /job>

Example showing job chain node parameters defined for the job chain

This example shows how job chain node parameters can be configured for a job chain and assigned to specific nodes in that chain, independently of the order being run or the jobs specified for individual nodes in the chain.

In addition, the example shows how these parameters can be called in shell jobs and used together with global variables.

Example Download

You can download the example:

• job_chain_node_params.zip

Instructions

• Unzip all files in the download file into the ./config folder of your JobScheduler installation.

• Define a global scheduler variable:

  • Open the config/scheduler.xml file and add the following lines before the <security> element:

    <params> 
    	<param name="global_test_var" value="def"/>
    </params>

• Restart the JobScheduler
• Open the JOC Cockpit in your browser
• Open the JOB CHAINS view.
• Navigate to the folder samples/configuration/
• Click Expand Details
• Find the job chain samples/configuration/chain_a.
• Start the order by choosing "Start order now" from the Order menu.
• Open the log  file

The order will run through both steps of the job chain. In each step the order parameters will be logged by the job. The parameters param1 - param4 should be shown in the log file with the following values:

State:first
       param1=10
       param2=abc
       param3=foo
       param4=abc---def

and


State:second
       param1=10
       param2=xyz
       param4=xyz---def

The next section explains how these parameters were configured.

How it works

The job chain chain_a consists of two steps, "first" and "second". Both steps use the same job readparam. The readparam job is a small JavaScript job which reads all order parameters and logs them on info-level.

The Java monitor implementation

The sos.scheduler.managed.configuration.ConfigurationOrderMonitor is configured as a monitor for the job. This is a special pre/post-execution script which is included in the JobScheduler distribution. It can be used for all jobs.

In the JobScheduler Object Editor, JOE, the monitor implementation is called under JobScheduler Objects.Jobs.Jobname.Pre-/Post-Processing as a Java configuration_monitor. The monitor reads special job chain parameter files which set parameters for a job chain and are able to configure each node of a job chain individually. These parameter files are named by default according to the pattern myJobChain.config.xml. See How to configure a Named Monitor for more information.

The screenshot below shows the monitor configuration in JOE.

The XML code generated by JOE for the readparam.job is:

<?xml version="1.0" encoding="ISO-8859-1"?>

<job  order="yes" stop_on_error="no">
    <params />

    <script  language="java:javascript">
        <![CDATA[
function spooler_process(){
      var orderparams = spooler_task.order.params;
      spooler_log.info("State: "+spooler_task.order.state());
      spooler_log.info("Params:");
      var paramNames = orderparams.names.split(";");      
      for( var i in paramNames ) {
        spooler_log.info( "  "+paramNames[i] + "=" + orderparams.value( paramNames[i] ) );
      }
      
      return true;
    }
        ]]>
    </script>

    <monitor  name="configuration_monitor" ordering="1">
        <script  language="java" java_class_path="" java_class="sos.scheduler.managed.configuration.ConfigurationOrderMonitor"/>
    </monitor>

    <run_time />
</job>

The following screen-shot shows the job chain node parameters for the example:

The next screen-shot shows the Details for JobChain form, which can be used to edit the param1 and param2 parameters. This form is opened with the Parameter button shown in the previous screenshot (top right):

Note that the node parameters - param3 and param4 are set and edited using JOE's XML editor, which is accessed using the OpenXML button shown in the screenshot.
(See also the Example in Detail section below.)

The example in detail:

The configuration monitor looks for the job chain parameter file in the current live folder and expects the name job_chain_name.config.xml. So in this case, the name of the job chain parameter file is chain_a.config.xml.

The full chain_a.config.xml file is listed in the next block. This file is opened and edited by clicking on the Open XML button in the job chain node parameter form (the Details for JobChain form) shown in the screenshot above.

<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" href="scheduler_configuration_documentation.xsl"?>
<settings>
  <job_chain name="chain_a">    
   <order>
     <params>       
       <param name="param1"                  value="10"/>
       <param name="param2"                  value="abc"/>                
     </params>
     <process state="first">
       <params>
         <!-- create parameter only for this state -->
         <param name="param3"                  value="foo"/>      
         <!-- use global and chain parameter in value of new parameter -->
         <param name="param4"                  value="${param2}---${global_test_var}"/>                
       </params>  
     </process>
     <process state="second">
       <params>
         <!-- overwrite param2 parameter -->
         <param name="param2"                  value="xyz"/>      
         <!-- use global and chain parameter in value of new parameter -->
         <param name="param4"                  value="${param2}---${global_test_var}"/>                
       </params>  
     </process>
   </order>   
  </job_chain>
</settings>

The chain_a.config.xml  file has three <params> elements:

• The first one is a child of the <order> element. It configures parameters for all steps of the job chain. That's why the log shows param1 as 10 in both steps.
  Parameters here were set in the job chain node parameter form (the Details for JobChain form) shown in the screenshot above.
• The second <params> element is a child of <process state="first">. It configures parameters for the first node (called "first") of the job chain. It sets two additional parameters param3 and param4. param4 uses a special ${paramname}-syntax to reference other parameters. It references param2 which was set in the first params element. As you see in the log, this part is later resolved to abc.
• The third is the global_test_var parameter that is a variable defined in config/scheduler_global_vars.xml.
  By configuring the global scheduler variable global_configuration_params in scheduler.xml to point to that file (scheduler_global_vars.xml), we tell the configuration monitor to always read that file and process it's parameters. In config/scheduler_global_vars.xml the value for global_test_var is set to def, so the reference in param4 is resolved to that value.

Note that:

• The second <params> element is a child of <process state="second">. It configures parameters for the second node (called "second") of the job chain. It overwrites the param2 parameter and sets it to xyz. param3 is not set at all, thus it doesn't appear in the log for state second. 
• param4 is defined in the same way as in state first, but this time the resulting value is different because param2 has been overwritten for this state.

Scope / Application

In job chain parameter files the ${PARAMNAME}-syntax can be used to reference:

• order parameters
• order parameters defined in the job chain parameter file
• global JobScheduler variables
• environment variables

Further References

Job Chain and Job Chain Node Parameters

• Page:
  Example for using node parameters in JITL Jobs
• Page:
  How to access node parameters when executing a job with a JobScheduler Agent
• Page:
  How to define and reference job chain node parameters
• Page:
  How to define job chain node parameters for orders
• Page:
  How to dynamically use node parameters and internal parameter substitution