How to manage a critical path for job processing

Scope

• JobScheduler implements the concept of a "critical path", i.e. the fact that in case of time constraints some less critical jobs should be skipped during daily or nightly processing.
• This article introduces the solution for JobScheduler.

Download & Installation

• Download the attached file: operations.zip
• Unzip the archive to your ./config/live/sos folder. All files contained will be unzipped to an operations sub-folder.

Explanations

The functionality includes job nodes to be skipped for critical path processing. This is achieved by

• marking a job node being part of a critical path by prefixing the job node state either with "+" (required) or "-" (skippable).
• running a new JITL job "activate_critical_path" that depending on its parameterization would run in one of these processing modes:
  • skip the job nodes that are marked as being skippable ("-") (default) or
  • skip the job nodes that are not marked as being required ("+").
• running a new JITL job "deactivate_critical_path" that would reverse the above operation.

The different processing modes are supported by the following configuration:

• Job node states in job chains can be marked to be skipped for critical path processing by prefixing them with a "-" character.
• Job node states in job chains can be marked to be required for critical path processing by prefixing them with a "+" character.

Usage UNDER REVIEW

A business process is comprises of 1 or more individual steps (jobs). A job may or many not need a precondition to start i.e. previous job must be successfully completed or its outcome doesn't affect the rest of the jobs in the job chain. such jobs are called optional or non critical jobs. Few common examples of such jobs are delete temporary files, create DB index or generate a XSL report. The  The critical path is series of steps(job) in a business process( job chain), which should be completed to achieve business process's goals i.e. complete transaction loading, generate Bills etc. 

Planned Critical Path Activation 

Scenario 

More often then not business process has a fix start time, average run time and a expected end time, but in case of circumstances  where fix start time has to be changed i.e. System maintenance, later then usual availability of input data etc.

• In such a exceptional situation to achieve the business process goals with adherence to the SLA, some jobs can be excluded from the business process (job chains).
• A non critical job can be marked in a job chain either "-" skippable or "+" critical job during job chain development.
• If a situation arises where part of the job chain has to be skipped, JobScheduler admin can start the  job activate_critical_path.job.xml. Job activate_critical_path.job.xml  will search through the JobScheduler configuration and set the non-critical job(s) as skip.
• A job chain with non-critical job(s) set to skip by all the Orders( manually/scheduled )
• Once business process completes only with  critical job, the job chains original state can be reset by the job deactivate_critical_path.job.xml

Example Job Chain

<?xml version="1.0" encoding="iso-8859-1"?>
<job_chain  orders_recoverable="yes" visible="yes" title="JobChainA" max_orders="3">
    <job_chain_node  state="Start" job="/sos/jitl/JobChainStart" next_state="-100" error_state="End_Err"/>
    <job_chain_node  state="-100" job="01_TaskA1" next_state="200" error_state="End_Err"/>
    <job_chain_node  state="200" job="01_TaskA2" next_state="-300" error_state="End_Err"/>
    <job_chain_node  state="-300" job="01_TaskA3" next_state="400" error_state="End_Err"/>
    <job_chain_node  state="400" job="01_TaskA4" next_state="500" error_state="End_Err"/>
    <job_chain_node  state="500" job="01_TaskA5" next_state="End_Suc" error_state="End_Err"/>
    <job_chain_node  state="End_Suc" job="/sos/jitl/JobChainEnd" next_state="Success" error_state="Error"/>
    <job_chain_node  state="End_Err" job="/sos/jitl/JobChainEnd" next_state="Error" error_state="Error"/>
    <job_chain_node  state="Error"/>
    <job_chain_node  state="Success"/>
</job_chain>

Dynamic Critical Path Activation 

Scenario 

Some business process are dependent on other business process's outcome or a incoming file. A job chain can be configured to monitor a directory and would start as soon as a file matching with the regular expression found in the directory. So for example usually file arrives every day before 05:00 am job chain finishes successfully at 09:30 AM, but due to unexpected circumstances if file arrives later then the usual time it may be desirable that even though file arrives late job chain should finish at its usual time.

• In such a exceptional situation job chain can have first node of the chain which checks the current time and if it finds late arrival of the file, triggers the job activate_critical_path.job.xml for specific job chain.
• Once process finishes successfully the last node of the job chain can start the job deactivate_critical_path.job.xml which will reset the job chain status back to normal with all the jobs set to executed. 

Example Job Chain

<?xml version="1.0" encoding="iso-8859-1"?>
<job_chain  orders_recoverable="yes" visible="yes" title="JobChainA" max_orders="3">
    <job_chain_node  state="Start" job="/sos/jitl/JobChainStart" next_state="-100" error_state="End_Err"/>
    <job_chain_node  state="activate_critical_path" job="/sos/operations/activate_critical_path" next_state="-100" error_state="End_Err"/>
    <job_chain_node  state="-100" job="01_TaskA1" next_state="200" error_state="End_Err"/>
    <job_chain_node  state="200" job="01_TaskA2" next_state="-300" error_state="End_Err"/>
    <job_chain_node  state="-300" job="01_TaskA3" next_state="400" error_state="End_Err"/>
    <job_chain_node  state="400" job="01_TaskA4" next_state="500" error_state="End_Err"/>
    <job_chain_node  state="500" job="01_TaskA5" next_state="End_Suc" error_state="End_Err"/>
    <job_chain_node  state="deactivate_critical_path" job="/sos/operations/deactivate_critical_path" next_state="End_Suc" error_state="End_Err"/>
    <job_chain_node  state="End_Suc" job="/sos/jitl/JobChainEnd" next_state="Success" error_state="Error"/>
    <job_chain_node  state="End_Err" job="/sos/jitl/JobChainEnd" next_state="Error" error_state="Error"/>
    <job_chain_node  state="Error"/>
    <job_chain_node  state="Success"/>
</job_chain>