How to manage complex conditions to check previous job executions and to start successor job chains

Scope

• Use Case

• Create Job Stream #1 

  • 1. Batch Job #1 runs only at 10pm. If the job fails to run successfully, is should be rerun up to a maximum 2 times.

  • 2. Batch Job #2 will run IF AND ONLY IF Job #1 completes successfully by 11pm and only if today is Wednesday.

  • 3. Batch Jobs #3 and #4 will run only if Job #2 ran successfully before 11.30pm and only if it’s before the 10th of the month (meaning the run date is < 10th OCT-2015 for example).

  • 4. Batch Job #5 will run after midnight and if Jobs #3 and #4 completed running within 45 minutes of their individual start times.

  • 5. Batch Job #6 will run only on Thursdays and Fridays of the week, and IF AND ONLY IF it’s after the 5th of the current month AND before 15th of the month (meaning the run date is > 5th OCT-2015 && < 15th OCT-2015 for example).

• Create Job Stream #2:

  • 6. Run Batch Job #7 all by itself in a separate Job Stream #2 during the 1st week of the 3rd month of every quarter in a calendar year.

• Combine the 2 Job Streams into a new Job Stream #3:

  • 7. Batch Job #8 and Batch Job #9 (in Job Stream #3) will be automatically started and running at the same time IF AND ONLY IF Job #6 from Job Stream #1 and Job #7 from Job Stream #2 ran successfully.

• Create Job Stream # 4:

  • 8. Run Batch Job #10 (in Job Stream #4) only after the successful completion of Job #8 and #9 in Job Stream #3.

• Hint
  • This use case is derived from a complex workflow implemented with Control-M®.

Solution

• Download: workflow_condition.zip
• Extract the archive to a folder in your JobScheduler installation named ./config/live/issues.
• The archive will extract the files to a sub-folder workflow_condition. 
• Note that you can store the sample files in any folder you like: however, you will then have to adjust the job parameters accordingly.

Pattern

Implementation

Components

• The solution makes use of Scripting with Pre-/Post-Processing Monitors and date calculation.
• The solution contains four job streams with dependencies:
  • job_stream1 and job_stream2 represent two job chains that have to be executed successfully to make job_stream3 proceed.
  • job_stream3 splits the processing into parallel jobs (job8, job9).
  • job_stream4 is the final job chain that checks if job8 and job9 from job_stream3 did complete successfully.
• The solution implements the following job dependencies:
  • job_stream1
    • job1
      • runs without dependencies.
      • is configured for setback, i.e. will repeat iself in case of error up to two times.
    • job2
      • implements an individual Pre-Processing Monitor to check if the current day is Wednesday and if the predecessor job job1 completed successfully before 11:00pm.
      • will be skipped and the order moved to the next job node if the check fails.
    • job3
      • implements an individual Pre-Processing Monitor to check if the current date is before the 10th of month if the predecessor job job2 completed successfully before 11.30pm.
      • will be skipped and the order moved to the next job node if the check fails.
    • job4
      • implements the same check as job3.
    • job5
      • implements an individual Pre-Processing Monitor to check if two predecessor jobs (job3, job4) completed successfully and consumed less than 45 min. execution time.
      • will be skipped and the order moved to the next job node if the check fails.
    • job6
      • implements an individual Pre-Processing Monitor to check if the current day is a Thursday or Friday and if the date is after the 5th and before the 15th of month.
      • makes use of the Named Post-Processing Monitor start_job_stream (by a <monitor.use> element):
        • this Monitor checks if a number of jobs completed successfully: 
          • the job names can be specified by use of the check_jobs job parameter that is assigned a semicolon separated list of job names. 
          • job names can be specified with an absolute path (starting from the live folder) or with a path relative to the directory of this job.
        • if the checks are successful then this Monitor will start a number of job chains; 
          • the job chain names can be specified by use of the start_job_chains job parameter that is assigned a semicolon separated list of job chain names.
          • job chain names can be specified with an absolute path (starting from the live folder) or with a path relative to the directory of this job.
      • will be skipped and the order moved to the next job node if the check fails.
  • job_stream2
    • job7
  • job_stream3
    • job8
    • job9
  • job_stream4
    • job10
• The solution implements a job check_predecessor_job_chain_Z that has been added at the start of job_chain_Z.
  • This job is parameterized with a number of parameters that start with the prefix "check_predecessor" and use an arbitrary suffix that is required in order to create unique parameter names. The value of each parameter is the final job of the respective job chain that is checked for successful execution:
    • Example:
      • check_predecessor_A = issues/final_chain/do_something_job_chain_A
      • check_predecessor_B = issues/final_chain/do_something_job_chain_B
    • Should the sample files not have been extracted to the ./config/live/issues/final_chain folder then please adjust the parameter value to point to the relevant directory.
  • This job implements a spooler_process()function that reads the parameters and checks the job history for successful execution of the jobs specified by the parameters.
  • Should all checks for previous successful execution provide a positive result then the current order will be moved to the next job chain node. Otherwise the current order is set back and will be repeated regularly according to the setback configuration for this job.

Usage

• Positive Check
  • Add an order to job_chain_A using JOC's Add Order context menu.
    • The order should pass through the job chain without errors.
  • Add an order to job_chain_B using JOC's Add Order context menu.
    • The order should also pass through the job chain without errors.
  • Add an order to job_chain_B using JOC's Add Order context menu.
• Negative Check
  • Modify the job do_something_job_chain_A to include an error, e.g. by adding the command exit 1 as the final line in the job script.
  • Add an order to job_chain_A:
    
    • The order should proceed with an error.
  • Add an order to job_chain_Z
    • The order should proceed with an error that is visible in the order log. This error will state that the predecessor job did not complete successfully.
    • The order will be set back and be repeated according to the setback intervals that have been specified for the job check_predecessor_job_chain_Z.
  • Removing the modification to job do_something_job_chain_A and adding an order to job_chain_A that then runs successfully should result in a successful run of job_chain_Z - either by a new order added to job_chain_Z or by the next execution of the previous order that has been set back.