Overview

This solution is about monitoring JobScheduler and its objects such as Jobs, Job Chains and Orders. Here you go an overview of the JobScheduler monitoring workflow:

These are some of the features of the architecture of this solution:

Definitions

Benefits

The benefits of the new solution are:

  1. There is no changes to be done in your JobScheduler configuration (Jobs, Job Chains, etc.) in order to get this solution working. You have to add the corresponding Job Chains for the monitoring but do not have to modify your current ones.
  2. The whole architecture lies at JobScheduler side and the solution is then independent from the monitor that the alerts are sent to. The solution works for every monitor that can receive passive checks.
  3. Processing of Jobs and Job Chains in JobScheduler is not affected or modified by the monitoring, neither in sense of performance nor in sense of stability.

Functionality

Installation and Configuration

As we have seen, the architecture lies at the JobScheduler side, therefore most of the installation and configuration has to be done at the JobScheduler side.

Database Tables

Three database tables have to be set at the JobScheduler database:

Java Program

The following JAR file has to be included in your \lib folder for JobScheduler:

com.sos.scheduler.notification-xxx.jar (xxx for the version number)

This JAR is currently not included within the JobScheduler installation.

XML Schemas

Schemas and XML files (see examples below) have to be placed together at \config\notification

Schema: CheckHistoryConfiguration_v1.0.xsd

Description:

  1. Specifies the JobScheduler that should be monitored
  2. Specifies the JobScheduler objects that should be monitored
  3. Timers check the job and job chain execution for timeouts

Example: CheckHistoryConfiguration.xml

Here you go an example of an XML file used for the monitoring of a specific JobChain:

   <?xml version="1.0" encoding="utf-8"?>
      <CheckHistoryConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="CheckHistoryConfiguration_v1.0.xsd">
	  <MonitoredObject>
		 <JobChains>
			<JobChain scheduler_id="MY_JOB_SCHEDULER_ID" name="MY_JOB_CHAIN_NAME"/>
		 </JobChains>
		<Timers>
                   <Timer>
			<JobChains>
			        
					<JobChain name="MY_JOB_CHAIN_NAME" />
			</JobChains>
               
                
			 	<Maximum><Script language="javascript"><![CDATA[
					function calculate()\{
						var fileSize				= new java.lang.Double(%FILE_SIZE%);
						var timerExpiryFactor 		        = 0.0025;
						var timerExpiryTolerance 	        = timerExpiryFactor*0.1;
						var timerExpiry 			= new java.lang.Double(timerExpiryFactor+timerExpiryTolerance);
						timerExpiry 				= timerExpiry*fileSize;
					return timerExpiry;
					\} 
					calculate();
				]]></Script></Maximum>
		      </Timer>
		</Timers>
          </MonitoredObject>
  </CheckHistoryConfiguration>

Following the description above:

  1. Specifies the JobScheduler that should be monitored: Job Scheduler with "MY_JOB_SCHEDULER_ID"
  2. Specifies the JobScheduler objects that should be monitored: Job Chain "MY_JOB_CHAIN_NAME"
  3. Timers check the job and job chain execution for timeouts: Timer for Job Chain "MY_JOB_CHAIN_NAME" (moreover a function that calculates the expiration time for the timer)

Schema: SystemMonitorNotification_v1.0.xsd

Description:

  1. Configuration for Notifications to a specific System Monitor
  2. Specifies the type of notification: "service_name_on_error" or "service_name_on_success"
  3. Specifies then the EXACT name of the service (the way it is named at the System Monitor)
  4. Specifies the EXACT hostname for the host the notification are sent from (the way it is named at the System Monitor)
  5. Specifies the port the application to receive passive checks is running on
  6. Specifies the hostname of the System Monitor, that is the hostname for the host the notification are sent from
  7. Define the type of encryption is used to send the information to the System Monitor
  8. Specifies how many notifications have to be sent to the System Monitor for a specific Job Scheduler object
  9. The same as above in case there is configured a Timer for this Job Scheduler object

Example SystemMonitorNotification_op5.xml

Here you go an example of an XML file used for notifying a specific System Monitor (op5 Monitor):

 <?xml version="1.0" encoding="utf-8"?>
  <SystemMonitorNotification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="SystemMonitorNotification_v1.0.xsd">
	<Notification>
	    <NotificationMonitor service_name_on_error="OMS Mass Processing Problem Tracking">
            <NotificationInterface service_host="OMS Interfacing Server" monitor_port="5667" monitor_host="dipsy.sos" monitor_encryption="XOR">order history id=%ORDER_HISTORY_ID%, job chain=%JOB_CHAIN_NAME%, order id=%ORDER_ID%, step =%ORDER_STEP_STATE%, error=%ERROR_TEXT%, check = %CHECK_TEXT%</NotificationInterface>
		</NotificationMonitor>
		
                  <NotificationObject>
			<JobChains>
				<JobChain scheduler_id="scheduler" notifications="20" name="MY_JOB_CHAIN_NAME"/> 
			</JobChains>
			<Timers>
				<Timer>
					<JobChains>
						<JobChain notifications="20" name="MY_JOB_CHAIN_NAME"/>
					</JobChains>
				</Timer>
			</Timers>
		  </NotificationObject>
	</Notification>
	
	<Notification>
	    <NotificationMonitor service_name_on_success="OMS Mass Processing History">
             <NotificationInterface service_host="OMS Interfacing Server" monitor_port="5667" monitor_host="dipsy.sos" monitor_encryption="XOR">order history id=%ORDER_HISTORY_ID%, job chain=%JOB_CHAIN_NAME%, order id=%ORDER_ID%</NotificationInterface>
		</NotificationMonitor>
		
                <NotificationObject>
			<JobChains>
				<JobChain scheduler_id="scheduler" notifications="20" name="MY_JOB_CHAIN_NAME"/>
			</JobChains>
		</NotificationObject>
	</Notification>
  </SystemMonitorNotification>

For this concrete example and following the description from above (about the schema):

  1. Configuration for Notifications to a specific System Monitor: op5 Monitor
  2. Specifies the type of notification: "service_name_on_error" or "service_name_on_success"
  3. Specifies then the EXACT name of the service (the way it is named at the System Monitor): "OMS Mass Processing Problem Tracking"
  4. Specifies the EXACT hostname for the host the notification are sent from (the way it is named at the System Monitor): "OMS Interfacing Server"
  5. Specifies the port the application to receive passive checks is running on: "5667"
  6. Specifies the hostname of the System Monitor, that is the hostname for the host the notification are sent from: "dipsy.sos"
  7. Define the type of encryption is used to send the information to the System Monitor: "XOR"
  8. Specifies how many notifications have to be sent to the System Monitor for a specific Job Scheduler object:"20" notfications for "MY_JOB_CHAIN_NAME"
  9. The same as above in case there is configured a Timer for this Job Scheduler object:"20" notfications for the Timer for "MY_JOB_CHAIN_NAME"

Job Chains

Job Chains for these solutions have to be placed under \live\notification. Four Job Chains were implemented for this solution and they have the following functions:

System Monitor

  1. The System Monitor receives just passive checks, that means, there are no active checks for monitoring JobScheduler. The only configuration here is the capability to receive passive checks from a remote host.
  2. The services in the System Monitor have to be in concordance with the JobScheduler configuration. Passive checks (services) have to be configured and named following the convention used in the XML described above for the JobScheduler (CheckHistoryConfiguration.xml and SystemMonitorNotification_op5.xml).

Use Cases

Recoverable Errors

Initial Situation: A Job Chain is triggered by directory monitoring. That is, when a certain file comes in a monitored folder, the Job Chain starts.

Problem: The Job Chain ended with error.

Handling: The System Monitor will be notified to the service related to the Job Chain with the message error. The error message at the System Monitor will stay till the same file is again placed in the monitored directory and the Job Chain ends without errors. That means, that a new file makes the Job Chain end without errors, does not mean that the error is recovered, since the file that has been processed is now another one.

Configuration:

Workflow Execution takes too long

Initial Situation: A Job Chain is triggered and it could not end, it hanged in a step, taking then longer than expected.

Problem: Execution time was too long

Handling: A timer for this Job Chain is set and the System Monitor will be notified about it. The expiration times for the Job Chains are configured with enough time for processing, that means, this is usually used for cases where the Job Chain hanged in a specific step.

Configuration:

SFTP connection refused

Initial Situation: There is a Job Chain that uses SFTP for transferring files. You have a setback configured in this step of the Job Chain, so that if the connection to the SFTP server fails, this step is retried after some time.

Problem: The SFTP server is not available anymore.

Handling: The System Monitor will be notified to the service related to the Job Chain with the message error. (to be completed)

Configuration:

Counting number of Workflow Executions

Initial Situation: A specific number of Workflow Executions have to be executed till some specific time. That is, a specific value has to be monitored in order to determine if this quote was reached.

Handling: A new service for History is configured, so that the workflow executions (Job Chains when talking about JobScheduler) send the information that they were executed and finished (with specific values if required) to the System Monitor.

Configuration: