Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Scope

  • Use Case
    • Run a specific job within a job chain in multiple parallel instances, i.e. split a job for parallel processing.
      • The use case is intended for execution of the same job within a job chain in parallel instances, not for parallel execution of different jobs.
      • Typically each job instance is focused, e.g. on a partition in an Oracle database. The job chain is intended to query database partitions in parallel.
    • Synchronize jobs after parallel processing
  • Solution Outline
    • The number of parallel job instances is dynamically determined by an initial job.
    • Subsequently the respective number of orders is created for a specific job that would be executed in parallel tasks. The number of parallel tasks can be restricted by a process class, forcing job instances to wait for a task to become free.
    • Finally the parallel job instances are synchronized for further processing.
  • References

Solution

  • Download parallel_job_instances.zip
  • Extract the archive to a folder ./config/live/issues of your JobScheduler installation.
  • The archive will extract the files to a folder parallel_job_instances. 
  • You can store the sample files to a any folder as you like, the solution does not make use of specific folder names or job names.

Pattern

Implementation

Components

  • The job chain and jobs job1 to job5 provided by the sample are not specifiic for this solution, they represent most simple shell scripts.
  • The job split_partitions is the Splitter JITL Job provided with the Java class com.sos.jitl.splitter.JobChainSplitterJSAdapterClass.
    • The job is used with the following parameters:
      • state_names: a list of semicolon separated job node states to which splitted orders are added. For each entry in this list an individual order is created.
        • The state names correspond to the states that the respective jobs are associated with in the job chain definition.
        • In order to support better graphical output for the JOE diagram view the state names are prefixed with the state of the split_partitions job (corresponding its name), a colon and the name of the job associated with the state. Example for state of job3: split_partitions:job3
        • Example for state_names parameter value: split_partitions:job3;split_partitions:job4
      • sync_state_name: this parameter is only required for improved graphical output from JOE. It assumes the value of the state that is associated with the sync_partitions job. 
    • Any number of jobs can follow the jobs that are referenced by the state_names parameter.
  • The job job_chain1.sync_partitions is used to synchronize splitted orders and is provided by the Sync JITL Job with the Java class com.sos.jitl.sync.JobSchedulerSynchronizeJobChainsJSAdapterClass.
    • The recommended naming convention for this job includes to use the job chain name as a prefix: due to the nature of this job to synchronize across all job chains that are running in the system and that make use of the same sync job name you should prefix the job chain name to the job name if you want this job to be limited to work for the current job chain.
    • This job is used without parameters.

Usage

  • Add an order to job chain job_chain1 by use of JOC.
  • Consider the processing that would 
    • split the execution into two subsequent orders that run for job3->job3a and job4->job4a. 
    • move the current order to the sync node.
  • The processing for job3->job3a and job4->job4a will require different execution time. All orders will wait in the sync_partitions job node for any splitted order to arrive and with splitted orders being complete the processing will continue with job5.

 

 

 

 

  • No labels