Example showing how to set up a join job

Scope

The Join Job (JobSchedulerJoinOrders) is used to join up two parallel executing child segments in a Job Chain. It then continues processing in a single thread once processing of the parallel threads has been completed. It is used in two Job Chain Patterns - Split and Join and "Y".

This article describes how the Join Job can be used in a "Y" pattern - the How to Execute Jobs in a Job Chain in Parallel article describes it use in a Split and Join pattern.

The Join Job is only available for JobScheduler versions 1.11.4 and newer. FEATURE AVAILABILITY STARTING FROM RELEASE 1.11.4

See the How To Synchronize Job Execution Across Job Chains article for information about synchronizing Jobs in different Job Chains and for joining up parallel executing child Job Chain segments in JobScheduler versions 1.11.3 and older.

Join Patterns

The example described in this article shows the use of a single instance of the Join Job within a single Job Chain. Multiple instances of the Join Job can also be used within a Job Chain and the Job can also be used across multiple Job Chains. See the Configuration section of the JobSchedulerJoinOrders documentation for more information.

split and join Pattern

The configuration of the Join Job for the Split and Join pattern is described in the How to Execute Jobs in a Job Chain in Parallel article.

Relevant for the current article is that in the Split and Join pattern, the Split Job generates the configuration information required by the Join job. This information must be configured separately for the "Y" pattern. The information generated by the the Split Job is:

a parameter - required_orders - which is forwarded to the Join Job. This parameter defines the number of orders that the Join Job has to receive before processing the main Job Chain continues with processing of the Join Job itself
an Order for each of the child Job Chain segments that is to be processed in parallel. The end state of each of these Orders is the state of the Join Job.

The ID set for each of the child Job Chain segment Orders is made up from the ID of the main Order and state in the Job Chain of the first Job in the child Job Chain segment.

'Y' Pattern

The following diagram shows the "Y"-pattern Job Chain used in the example download archive that is linked from this article. Note that some of the elements in this pattern have been given particular functions to provide a working example of the Join Job and are not required for its operation in "normal" use.

Solution

Download the example y_join.zip.
Extract the archive to a folder ./config/live of your JobScheduler Master installation.
The archive will extract the files to a folder y_join.
The y_join folder can be renamed if required, the solution does not require the use of specific folder or Job names.

Example Description

The Join Job has basically a counting function. This makes it significantly faster than the Sync Job which checks the IDs of the jobs being processed.

Note that some of the elements in this example have been given particular functions to demonstrate the functioning of the Join Job and are not required for its operation in "normal" use.

A Parent Order (in this example main_order) is started with a parameter (required_orders).

The required_orders parameter is read immediately by the Join Job, which will then wait until the number of Orders specified in the parameter (here it is 12) have been processed.
- Note that the Join Job only counts orders that have the state of the join Job as their end state (here join).
The main_order moves to the generate_orders Job, which generates a total of 10 Child Orders.
- This number is specified in a second parameter for the main_order - generate_orders.
- 5 of these child orders start processing at node 150 and 5 will start at node 160. All the Child Orders will end at the Join Job.
  - It is not important whether or not Child Orders take the same branch of the Job Chain as the main_order.
- All these Child Orders will be counted towards the required_orders parameter.
- Child Orders are given an Id based on the Parent Order ID plus a string.
  - The first generated Child Order in the example will be given the ID main_order_0.
The main_order itself then moves to the wait Job where it waits for a time specified in the Order's wait_time parameter (here 35 seconds).
- The sole purpose of this delay is to demonstrate that the main_order can reach the Join Job after the other Orders.
- This delay is not necessary for the functioning of the Join Job and the example will work with the wait_time parameter set to its minimum value of 1 (0 will cause an error).
When the main_order reaches the Join Job it will be counted towards the number of Orders specified in the required_orders parameter, making a total of 11 after all the generated Child Orders have been completed.
- Note that the main_order is the only Order that will be counted that does not have to have the state of the Join Job as its end state.
The main_order will now be suspended at the Join Job (without the Job being processed) until:
- a further Order that has the state of the Join Job as its end state is completed.

The main_order_add-order Order can now be used to increase the the total number of Orders counted by the Join Job by 1.

In the current example, running the main_order_add-order Order once will cause the number of Orders counted to reach the value set in the required_orders parameter (12).
- The join Job will now process the main_order which will then proceed along the Job Chain - in this example to the Job C with the state 200.
The ID of this Order has to follow the convention used for other Child Orders - that is, the ID of the parent Order plus a string.
- Note that this string may not contain an underscore character ("_").

Note that Child Orders such as the generated Orders or the manually configured the main_order_add-order Order in this example will only be recognized as such when they are started after the Parent Order has been started.

The Job Chain, Jobs and Orders

The two "Y" branches in this example and are shown in the pattern diagram above. Each branch has a different purpose:

The left branch is used by an Order (main_order) that proceeds over the Join Job and Job B to the end of the Job Chain (state: success).
The right hand branch is used by an Order (main_order_add-order) that starts at Job A and ends at the Join Job.

To run the example, start the main_order Order.

The y_join Job Chain

<?xml version="1.0" encoding="ISO-8859-1"?>
<job_chain  title="Y Join">
    <job_chain_node  state="100" job="generate_orders" next_state="wait" error_state="error"/>
    <job_chain_node  state="150" job="job_a" next_state="join" error_state="error" delay="10"/>
    <job_chain_node  state="wait" job="wait" next_state="join" error_state="error"/>
    <job_chain_node  state="join" job="join" next_state="200" error_state="error"/>
    <job_chain_node  state="200" job="job_b" next_state="success" error_state="error"/>
    <job_chain_node  state="success"/>
    <job_chain_node  state="error"/>
</job_chain>

The generate_orders Job contains the following script:

The generate_orders Job

<?xml version="1.0" encoding="ISO-8859-1"?>

<job  order="yes" stop_on_error="no">
    <script  language="java:javascript">
        <![CDATA[
function spooler_process(){

    // merge parameters from task and order
    var params = spooler_task.params;
    params.merge( spooler_task.order.params );
 
    // set variable
    var generate_orders = params.value( 'generate_orders' );
    var jobChain = spooler.job_chain('/test/join/y_join/y_join');

    // log parameter
    spooler_log.info( 'generate_orders = ' + generate_orders);
 
    // generate orders
    for (i=0;i<generate_orders;i++){
       var order = spooler.create_order();
       order.id = spooler_task.order.id + "_" + i;
       order.params.merge(spooler_task.order.params);
       order.end_state="join";
       order.state="150";
       jobChain.add_order(order);
    }

    return true;
}
        ]]>
    </script>

    <run_time />
</job>

The test Order starts the first Job (job_create_orders) in the job_chain_test Job Chain:

this Job contains a script that generates the orders (see line XX of the listing) for the branch of the Job Chain from the job_nix Job to the join Job.

As the end_state for the generated Orders is the join Job state, these orders will be registered by the join job and counted towards to the value set by the job_create_orders shell script in the for loop (10)

The Orders

The Main Order

The main Order in this example has 3 parameters:

required_orders
- This parameter (Default 12.)
wait_time
- this is the time in seconds that the Wait Job will wait before the main Order moves to the Join Job. (Default 35 secs.)
- This parameter is not required by the Join Job.
generate_orders
- this is the number of Orders that are to be generated by the generate_orders Job. (Default 10.)
- This parameter is not required by the Join Job.

The Child Orders

The main_order_add-order Order

This Order is configured with:

state = 150 (The state in the Job Chain where the Order starts processing. Here this corresponds to job_a) and
end_state = join (The state corresponding to the Join Job) This means that this Order will be registered by the Join Job as counting towards the required orders.

.....

The main_order Order

<?xml version="1.0" encoding="ISO-8859-1"?>
<order >
    <params >
        <param  name="required_orders" value="12"/>
        <param  name="wait_time" value="35"/>
        <param  name="generate_orders" value="10"/>
    </params>
    <run_time />
</order>

.....

The main_order_add-order Order

<?xml version="1.0" encoding="ISO-8859-1"?>
<order  state="150" end_state="join">
    <run_time />
</order>

.....

Logging

A parameter can be set for the Join Job - show_join_order_list. When this parameter is set to true the all the Child Orders counted by the join job will be listed in the Parent Order log file.

The default setting for this parameter is false.

Space shortcuts

Page tree

Scope

Join Patterns

split and join Pattern

'Y' Pattern

Solution

Example Description

The Job Chain, Jobs and Orders

The Orders

The Main Order

The Child Orders

Logging