Product Knowledge Base

Space shortcuts

Page tree

Versions Compared

Old Version 8

changes.mady.by.user Alan Amos

Saved on

compared with

New Version Current

changes.mady.by.user Alan Amos

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Minor changes to 'Scope'

Table of Contents

Anchor
scope
scope
Scope

The Join Orders Job (JobSchedulerJoinOrders) is a JITL Job and is used to join up two parallel executing child segments in a Job Chain. It then continues processing with a single Order in a single thread once processing of the parallel threads has been completed. It is used in two Job Chain Patterns - Split and Join and "Y".

This article describes how the Join Orders Job can be used in a "Y" pattern , - the How to Execute Jobs in a Job Chain in Parallel article describes it its use in a Split and Join pattern.

The Join Orders Job is only available for JobScheduler versions 1.11.4 and newer.

Display feature availability
StartingFromRelease1.11.4

Join Patterns

  • The

...

  • Join Orders Job provides a plug-in replacement for the Sync Job for sync/join operations within a Job Chain

...

  • . As the

...

  • Join Orders Job is significantly faster than the Sync Job, users of JobScheduler 1.11.4 or newer wishing to improve performance of the Sync Job can simply replace their Sync Job with the Join Orders Job.

Join Patterns

The example described in this article shows the use of a single instance of the Join Orders Job within a single Job Chain. Multiple instances of the Join Orders Job can also be used within a Job Chain. See the Configuration section of the JobSchedulerJoinOrders documentation for more information.  

split and join Pattern

The configuration of the Join Orders Job for use in the Split and Join pattern is described in the How to Execute Jobs in a Job Chain in Parallel article.

Relevant for the current article is that in the Split and Join pattern, the Split

split and join Pattern

The configuration of the Join Job for the Split and Join pattern is described in the How to Execute Jobs in a Job Chain in Parallel article.

Relevant for the current article is that in the Split and Join pattern, the Split Job generates the configuration information required by the Join Orders job. This information must be configured separately for the "Y" pattern. The information generated by the the Split Job is:

  • a parameter - required_orders - which is forwarded to the Join Job. This parameter defines the number of orders that the Join Orders Job has to receive before processing the main Job Chain continues with processing of the Join Orders Job itself
  • an Order for each of the child Job Chain segments that is to be processed in parallel. The end state of each of these Orders is the state of the Join Orders Job.

The ID set for each of the child Job Chain segment Orders is made up from the ID of the main Order and state in the Job Chain of the first Job in the child Job Chain segment.

'Y' Pattern

'Y' Pattern

The following diagram shows the "Y"-pattern Job Chain used in the The following diagram shows the "Y"-pattern Job Chain used in the example download archive that is linked from this article. Note that some of the elements in this pattern have been given particular functions to provide a working example of the Join Job and are not required for its operation in "normal" use.

Flowchart
main
Flowchart
main [label="Order:main_order",fillcolor="green"]
generate100 [label="generate orders",fillcolor="lightskyblue"]
add [label="Order:main_order_add-order",fillcolor="green"]
wait150 [label="wait",fillcolor="lightskyblue"]
150200 [label="150200 job_a",fillcolor="lightskyblue"]
160300 [label="160300 job_b",fillcolor="lightskyblue"]
join [label="join orders",fillcolor="lightskyblue"]
200400 [label="200400 job_c",fillcolor="lightskyblue"]
success [label="success",fillcolor="orange"]

main -> 100 generate
generate100 -> wait150 
wait150  -> 150200 
150200 -> join 
add -> 160300 
160300 -> join 
join -> 200400 
200400 -> success

Solution

  • Download the example y_join.zip
  • Extract the archive to a folder ./config/live of your JobScheduler Master installation.
  • The archive will extract the files to a folder y_join. 
  • The y_join folder can be renamed if required, the solution does not require the use of specific folder or Job names.

Implementation

The Join Job has basically a counting function. This makes it significantly faster than the Sync Job which checks the IDs of the jobs being processed.

Note that some of the elements in this example have been given particular functions to demonstrate the functioning of the Join Job and are not required for its operation in "normal" use.

A Parent Order (in this example main_order) is started with a parameter (required_orders).

  • Note that the generate and wait Jobs in this example have been included to make the example work and are not required for its operation in "normal" use.
  • All Jobs in the "Y"-pattern are configured as a single Job Chain.
  • The parallel arms of the example are effectively job_a to the Join Orders Job and job_b to the Join Orders Job.  

Download and Configure the Example

  • Download the example y_join.zip.
  • Extract the archive to a folder ./config/live of your JobScheduler Master installation.
  • The archive will extract the files to a folder y_join. 
  • The y_join folder can be renamed if required, the solution does not require the use of specific folder or Job names.
  • Change the path to the example Job Chain in line 14 of the generate_orders Job below to the correct setting.

Example Description

To use the example, first start the Parent Order (in this example main_order). This Order has been configured with three parameters, one of which - required_orders - is required by the Join Orders Job. The other two parameters - generate_orders and duration_wait_job are only used to provide a convenient way of making the example work.

  • The required_orders parameter is read immediately by the Join Orders Job, which will then wait in the suspended state The required_orders parameter is read immediately by the join Job, which will then wait until the number of Orders specified in the parameter (here it is 1210) have been processed.
    • Note that the Join Job only counts orders that have the state of the join Job as their end state (here join).
  • The main_order moves to the generateThe main_order moves to the generate_orders Job, which generates a total of 10 8 Child Orders.
    • This number is specified in a second parameter for the main_order - generate_orders.
    • 5 4 of these child orders start processing at node 150 and end at the Join Job and 5 start at node 160 and be processed in one arm of the "Y" - i.e. starting at node 200 - and 4 will be processed in the other arm - i.e. starting at node 300. All the Child Orders will end at the Join Job.
      • It is not important whether or not Child Orders take the same The branch of the Job Chain as the main_orderthe Child Orders have been executed on is not important for the Join Orders Job.
    • All these Child Orders will be counted towards the required_orders parameter.
    • Child Orders are given identified by the Join Orders Job when they either:
      • Have an Id based on the Parent Order ID plus an underscore plus a string.
        The 
            • For example, the first generated Child Order in the example will be given the ID main_order_0.
            • This is the method used in the example.
          •  They carry a join_session_id parameter, specifying the ID of the Parent Order.
      • The main_order itself then moves to the wait Job where it waits for a time specified in the main_order's duration_wait_job parameter (here 30 itself then moves to the wait Job where it waits for a time specified in the Order's wait_time parameter (here 35 seconds).
        • The sole purpose of this delay is to demonstrate that the main_order can reach the Join Orders Job after the other Child Orders.
        • This delay is not necessary for the functioning of the Join Orders Job and the example will work with the the duration_wait_timejob parameter set to its minimum value of 1 (0 will cause an error).
      • When the main_order reaches the Join Orders Job it will be counted towards the number of Orders specified in the required_orders parameter, making a total of 11 of  8 + 1 = 9 Orders after all the generated Child Orders have been completed.
        • Note that the main_order is the only Order that will be counted that does not have to have the state of the Join Orders Job as its end state.
      • The main_order will now be suspended at the Join Orders Job (without the Job being processed) until:
        • a further Order that has the state of the Join Orders Job as its end state is completed. 

      The main_order_add-order Order can now be used to increase the the total number of Orders counted by the join Join Orders Job by 1.

      • In the current example, running the main_order_add-order Order once will cause the number of Orders counted to reach the value set in the required_orders parameter (10).
        • The join Join Orders Job will now process the main_order which will then proceed along the Job Chain - in this example to the Job C with the state 200400.
      • The ID of this Order has to follow one of the convention conventions used for other to identify Child Orders - that is, . Here the ID of the parent Order

      The Job Chain, Jobs and Orders

      The two "Y" branches in this example and are shown in the pattern diagram above. Each branch has a different purpose:

      • The left branch is used by an Order (main_order) that proceeds over the Join Job and Job B to the end of the Job Chain (state: success).
      • The right hand branch is used by an Order (main_order_add-order) that starts at Job A and ends at the Join Job.
      • plus an underscore plus a string has been used.
        • Note that this string may not contain an underscore character ("_") and therefore the string "add-order" has been used.

      Note that Child Orders such as the generated Orders or the manually started main_order_add-order Order in this example will only be recognized as such when they are started after the Parent Order has been started.

      The Job Chain, Jobs and Orders

      The Job Chain

      The Job Chain is shown in the diagram near the top of this article and is listed in the code block below.To run the example, start the main_order Order. 

      Code Block
      languagexml
      titleThe y_join Job Chain
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>

<job_chain  title="Y Join">
    <job_chain_node  state="100" job="generate_orders" next_state="wait150" error_state="error"/>
    <job_chain_node  state="150" job="job_await" next_state="join200" error_state="error" delay="10"/>
    <job_chain_node  state="wait200" job="waitjob_a" next_state="join" error_state="error"/>
    <job_chain_node  state="join300" job="joinjob_b" next_state="200join" error_state="error" delay="10"/>
    <job_chain_node  state="200join" job="job_bjoin" next_state="success400" error_state="error"/>
    <job_chain_node  state="success400"/>
    <job job="job_c" next_state="success" error_state="error"/>
    <job_chain_node  state="success"/>
    <job_chain_node  state="error"/>
</job_chain>

      The

      ...

      Jobs

      The Join Orders Job

      The Join Orders Job basically counts incoming Jobs. This makes it significantly faster than the Sync Job mentioned in the Scope section above, which checks the incoming Jobs for completeness.

      The configuration of the Join Orders Job can set using the JITL Job Wizard in JOE and is shown in following code block. Relevant for users in the following listing is the show_join order list parameter which may be optionally set (default is false). Setting this Parameter to true causes a list of all the orders counted by the Join Orders Job to be written to the Parent Order log file.

      Code Block
      languagexml
      titleThe Join Orders Job
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>

<job  order="yes" stop_on_error="no" idle_timeout="60" title="Join Job">
    <params >
        <param  name="show_join_order_list" value="true"/>
    </params>
    <script  language="java" java_class_path="" java_class="com.sos.jitl.join.JobSchedulerJoinOrdersJSAdapterClass"/>
    <run_time />
</job>

      Note that the Join Orders Job only counts Orders that have the state of the Join Orders Job as their end state (here join).

      Anchor
      generate_orders
      generate_orders
      The generate_orders Job

      The configuration of the generate_orders Job is shown in the next code block along with the script responsible for the generation of the Child Orders.

      Note that this Job is only used to create the current example and is not required for the Join Orders Job itself. In a working scheduling environment, the orders for the parallel parts of the "Y" would come from other sources such as Schedules or File Order Sources.

      Note also that the Job Chain path (set in line 14 of the script) must be modified to suit the actual path used.

      Code Block
      languagexml
      titleThe generate_orders Job
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>

<job  order="yes" stop_on_error="no">
    <script  language="java:javascript">
        <![CDATA[
function spooler_process(){

    // merge parameters from task and order
    var params = spooler_task.params;
    params.merge( spooler_task.order.params );
 
    // set variable
    var generate_orders = params.value( 'generate_orders' );
    var jobChain = spooler.job_chain('/test/join/y_join/y_join');

    // log parameter
    spooler_log.info( 'generate_orders = ' + generate_orders);
 
    // generate orders
    for (i=0;i<generate_orders;i++){
       var order = spooler.create_order();
       order.id = spooler_task.order.id + "_" + i;
       order.params.merge(spooler_task.order.params);
       order.end_state="join";
       if((i%2)==1) {
          order.state="200";
       } else {
          order.state="300";
       }
       jobChain.add_order(order);
    }

    return true;
}
        ]]>
    </script>

    <run_time />
</job>

      The Order starts the first Job (generate) in the y_join Job Chain:

      • This Job contains a script that generates the Child Orders (see lines 20 - 31 of the listing).
        • The Orders are alternated between the two branches of the Job Chain (even numbered Orders start at the Job corresponding with the Order state 150 and odd numbered Orders at the Job corresponding with the Order state 160). All Child Orders terminate at the Join Orders Job.
        • The total number of orders generated is determined by the generate_orders parameter.
      The wait Job

      The wait Job is configured to read the duration_wait_job parameter and execute a simple script (i.e. ping local host for the length of time specified in the parameter). This script causes the Job to wait for the number of seconds specified in the parameter.

      The Orders

      The Parent Order

      The Parent Order, in this example, with ID main_order has the following 3 parameters:

      • required_orders
        • This parameter (Default 10.)
        • This parameter is required for the Join Job.
      • duration_wait_job
        • this is the time in seconds that the wait Job will wait before the main Order moves to the Join Orders Job. (Default 30 secs.)
        • The duration_job_a parameter is only used in the wait Job as part of this example and is not necessary for the functioning of the Join Orders Job.
      • generate_orders
        • this is the number of Orders that are to be generated by the generate_orders Job. (Default 8.) 
        • The generate_orders parameter is used in the Generate Job as part of this example to specify the number of Child Orders that should be generated. This parameter is not necessary for the functioning of the Join Job as the Jobs counted by the Join Job could come from any number of sources..

      Code Block
      languagexml
      titleThe main_order Order
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>
<order >
    <params >
        <param  name="required_orders" value="10"/>
        <param  name="duration_job_a" value="30"/>
        <param  name="generate_orders" value="8"/>
    </params>
    <run_time />
</order>

      The Child Orders

      The generated Child Orders

      Have the ID of the Parent Order (main_order) + "_" + * where * is a string - in the current example simple numbers are used a string.

      Have either:

      • <order state="200" end_state="join"> or
      • <order state="300" end_state="join">

      depending on which branch of the Job Chain they should be executed on.

      The main_order_add-order Order

      Has the ID of the Parent Order (main_order) + "_" + * where * is a string - in the current example "add-order" is used.

      This Order is configured with:

      • state = 300 (The state in the Job Chain where the main_order_add-order Order starts processing. Here this is job_b) and
      • end_state = join (The state corresponding to the Join Job) This means that this Order will be registered by the Join Job as counting towards the required orders.
      Code Block
      languagexml
      titleThe generate_orders Job
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>

<job  order="yes" stop_on_error="no">
    <script  language="java:javascript">
        <![CDATA[
function spooler_process(){

    // merge parameters from task and order
    var params = spooler_task.params;
    params.merge( spooler_task.order.params );
 
    // set variable
    var generate_orders = params.value( 'generate_orders' );
    var jobChain = spooler.job_chain('/test/join/y_join/y_join');

    // log parameter
    spooler_log.info( 'generate_orders = ' + generate_orders);
 
    // generate orders
    for (i=0;i<generate_orders;i++){
       var order = spooler.create_order();
       order.id = spooler_task.order.id + "_" + i;
       order.params.merge(spooler_task.order.params);
       order.end_state="join";
       order.state="150";
       jobChain.add_order(order);
    }

    return true;
}
        ]]>
    </script>

    <run_time />
</job>

      The test Order starts the first Job (job_create_orders) in the job_chain_test Job Chain:

      • this Job contains a script that generates the  orders (see line XX of the listing) for the branch of the Job Chain from the job_nix Job to the join Job.

       

      As the end_state for the generated Orders is the join Job state, these orders will be registered by the join job and counted towards to the value set by the job_create_orders shell script in the for loop (10)

      The Orders

      The Main Order

      The main Order in this example has 3 parameters:

      • required_orders
        • This parameter (Default 12.)
      • wait_time
        • this is the time in seconds that the Wait Job will wait before the main Order moves to the Join Job. (Default 35 secs.)
        • This parameter is not required by the Join Job.
      • generate_orders
        • this is the number of Orders that are to be generated by the generate_orders Job. (Default 10.) 
        • This parameter is not required by the Join Job.

      The Secondary Orders

      The main_order_add-order Order

      This Order is configured with:

      • state = 150 (the state corresponding to job_a) and
      • end_state = join (the state corresponding to the Join Job) This means that this Order will be registered by the Join Job as counting towards the required orders.

       

      .....

      Code Block
      languagexml
      titleThe main_order Order
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>
<order >
    <params >
        <param  name="required_orders" value="12"/>
        <param  name="wait_time" value="35"/>
        <param  name="generate_orders" value="10"/>
    </params>
    <run_time />
</order>

      ...

      Code Block
      languagexml
      titleThe main_order_add-order Order
      linenumberstrue
      collapsetrue
      <?xml version="1.0" encoding="ISO-8859-1"?>
<order  state="150300" end_state="join">
    <run_time />
</order>

      Logging

      A parameter can be set for the Join Job - show_join_order_list. ...When this parameter is set to true the all the Child Orders counted by the join job will be listed in the Parent Order log file.

      The default setting for this parameter is false.