Scope
- Use Case
- Run a specific job within a job chain in multiple parallel instances, i.e. split a job for parallel processing.
- The use case is intended for execution of the same job within a job chain in parallel instances, not for parallel execution of different jobs.
- Typically each job instance is focused, e.g. on a partition in an Oracle database. The job chain is intended to query database partitions in parallel and to be parameterized individually.
- Synchronize jobs after parallel processing
- Run a specific job within a job chain in multiple parallel instances, i.e. split a job for parallel processing.
- Solution Outline
- The number of parallel job instances is dynamically determined by an initial job.
- Subsequently the respective number of orders is created for a specific job that would be executed in parallel tasks. The number of parallel tasks can be restricted by a process class, forcing job instances to wait for a task to become free.
- Finally the parallel job instances are synchronized for further processing.
- References
Solution
- Download parallel_job_instances.zip
- Extract the archive to a folder
./config/live/issues
of your JobScheduler installation. - The archive will extract the files to a folder
parallel_job_instances.
- You can store the sample files to a any folder as you like, the solution does not make use of specific folder names or job names.
Pattern
Implementation
Components
- The job chain provided by the sample is not specifiic for this solution, the same applies to the
query
job.. - The
prepare
job is used to dynamically create the number of parallel instances of thequery
job. In the sample this job creates an order parameterparallel_executions=5
for use with the subsequentgenerate
job. - The
generate
job implements the solution by creating the number of orders specified by theparallel_executions
parameter.- Each order as added two parameters
number_of_orders
: a copy of theparallel_executions
parameter.order_sequence
: the sequence number of the order created. The sequence starts from 1 up to the value of theparallel_executions
parameter.
- The orders are assigned the state that is associated with the
query
job, i.e. the orders will be executed starting with that state. - The orders are assigned the end state that is associated with the
synchronize
job.
- Each order as added two parameters
- The
query
job is configured for a maximum number of 3 parallel tasks via the attribute<job tasks="3">
. It could be configured for any number of parallel tasks. For the sake of this sample the limitation shows the behavior of the job to wait for processes to become free that could be assigned to subsequent orders for the same job. - The job
synchronize
is used to synchronize splitted orders and is provided by the Sync JITL Job with the Java classcom.sos.jitl.sync.JobSchedulerSynchronizeJobChainsJSAdapterClass
.- This job is used without parameters.
Usage
- Add an order to job chain
parallel_execution
by use of JOC. - Consider the processing that would
- split the execution into 9 subsequent orders that run for the
query
job. - move the current order to the
synchronize
job node.
- split the execution into 9 subsequent orders that run for the
- The splitted orders for the
query
job will arrive in thesynchronize
job node and will wait for all orders to be completed. With all splitted orders being completed the processing will continue with jobdo_something
.