JobScheduler Monitoring Interface - Feature Documentation

Overview

This solution is about monitoring JobScheduler and its objects such as Jobs, Job Chains and Orders. Here you get an overview of how JobScheduler monitoring works. This feature will be available starting from general availability release 1.8.

These are some of the features of the architecture of this solution:

• JobScheduler: The architecture establishes a partition between:
  • Detecting errors: A Job Chain analyses the JobScheduler logging and checks whether the monitored Job Scheduler objects had errors or warnings.
  • Sending alerts: Another Job Chain is responsible for sending the alerts to the corresponding System Monitor. The difference here, is that not all alerts are only incidents, but also events, as in occurrences, for example, the alert that a specific Job Chain was executed and which result it ended up with.
• JobScheduler: This architecture allows to analyse the Log History of more than one JobScheduler.
• System Monitor: JobScheduler is able to connect to more than one System Monitor at the same time.

Definitions

• System Monitor: A System Monitor is an instrument to inform the Service Desk (1st Level Support) about incidents in IT systems. It does not serve for the analysis of the incidents, but merely for the information about the incidents, in order to be able to forward and scale these informations.

• Passive Checks: These kind of checks are the ones that are sent remotely from an external host (from the point of view of a System Monitor) to the System Monitor. Otherwise, the ones that are carried out periodically by the System Monitor are called active checks.

• Alerting: An Alert is an alarm, i.e. the message about an event. An alert does not provide every relevant information of an event, but it informs about the existence of the event. An alert can be either positive or negative.

• Notification: The notification of a specific alert. Not every alert will be notified, just the ones that are so configured will be notified. Notifications are therefore a subset of the alerts and can be either positive or negative too.

• Acknowledgement: Is the confirmation of an alert and it has the meaning, that the alert has been seen and/or is well known and the incident is trying to be recovered. An acknowledgement is always manually executed, that means, there is always someone that has realized there is a Critical service and this person acknowledges the services (usually by the Service Desk or 1st Level Support). It is never an automatized step.

Benefits

The benefits of the new solution are:

1. There is no changes to be done in your JobScheduler configuration (Jobs, Job Chains, etc.) in order to get this solution working. You have to add the corresponding Job Chains for the monitoring but do not have to modify your current ones.
2. The whole architecture lies at JobScheduler side and the solution is then independent from the monitor that the alerts are sent to. The solution works for every monitor that can receive passive checks.
3. Processing of Jobs and Job Chains in JobScheduler is not affected or modified by the monitoring, neither in sense of performance nor in sense of stability.
4. The level of detail in a message of a Service in the System Monitor is much higher with this solution. JobScheduler logs very exact what the error is about and this information is sent as a passive check to the specific Service, which shows the log message that JobScheduler logged.
5. The criticality of an error is immediately recognized in the System Monitor. JobScheduler has all information about errors and this information is sorted out and sent to different Services in the System Monitor for every specific case. Through this feature, the Service Desk is immediately able to set its priority for recovering errors. For example, it does not have the same Criticality to recover an error of Performance (low) than when Documents could not be generated (high). Here you go a representation of this feature:

Functionality

Installation

See https://kb.sos-berlin.com/x/fYEm

Configuration

JobScheduler - SystemMonitorNotification files

Location: <scheduler_install>/config/notification

SystemMonitorNotification Elements

The configuration element descriptions are organized into the following major categories:

SystemMonitorNotification

SystemMonitorNotification support the following attributes:

Note:

• attribute system_id 

• in case of the SystemMonitorNotificationTimers.xml the value of this attribute is not important and can have any value.

  • e.g.: timers

<SystemMonitorNotification system_id="OP5">
...

SystemMonitorNotification / Notification

The following elements may be nested inside a Notification element:

SystemMonitorNotification / Notification / NotificationMonitor

NotificationMonitor support the following attributes:

Note:

• attributes service_name_on_error and service_name_on_success
  • at least one of these attributes must be configured
  • both attributes can be configured together

<!--
Example OP5
NSCA Status:
0 - OK
1 - WARNING
2 - CRITICAL
3 - UNKNOWN
-->
...
<!--
Sending occurred errors as CRITICAL (default)
-->
<NotificationMonitor service_name_on_error="JobScheduler Errors">
...
<!--
Sending occurred errors as WARNING
-->
<NotificationMonitor service_name_on_error="JobScheduler Errors" service_status_on_error="1">
...        

One of the following elements must be nested inside a NotificationMonitor element:

SystemMonitorNotification / Notification / NotificationMonitor / NotificationInterface

NotificationInterface support the following attributes:

...
<NotificationInterface monitor_host="monitor_host" monitor_port="5667" monitor_encryption="XOR" service_host="service_host"><![CDATA[
scheduler id=%MON_N_SCHEDULER_ID%, history id=%MON_N_ORDER_HISTORY_ID%, job_chain=%MON_N_JOB_CHAIN_NAME%(%MON_N_ORDER_ID%), step =%MON_N_ORDER_STEP_STATE%, error=%MON_N_ERROR_TEXT%
]]></NotificationInterface>
...

SystemMonitorNotification / Notification / NotificationMonitor / NotificationCommand

NotificationCommand support the following attributes:

...
<NotificationCommand><![CDATA[
echo scheduler id=%MON_N_SCHEDULER_ID%, history id=%MON_N_ORDER_HISTORY_ID%, job_chain=%MON_N_JOB_CHAIN_NAME%(%MON_N_ORDER_ID%), step =%MON_N_ORDER_STEP_STATE%, error=%MON_N_ERROR_TEXT% > D://errors.txt
]]></NotificationCommand>
...

SystemMonitorNotification / Notification / NotificationObjects

One of the following elements must be nested inside a NotificationObjects element:

SystemMonitorNotification / Notification / NotificationObjects / JobChain

JobChain support the following attributes:

...
<JobChain notifications="2" name="test/my_jobchain"/>
...
<JobChain scheduler_id="scheduler_4444" />
...
<JobChain scheduler_id="scheduler_4444" name="^(test/my)" />
...
<JobChain name="test/my_jobchain" step_from="200"/>
...
<JobChain name="test/my_jobchain" step_to="500"/>
...
<JobChain name="test/my_jobchain" step_from="300" step_to="300"/>
...
<JobChain name="test/my_jobchain" excluded_steps="200;300"/>
...

SystemMonitorNotification / Notification / NotificationObjects / Timer

Timer support the following attributes:

SystemMonitorNotification / Timer 

The following elements must be nested inside a Timer element:

Timer support the following attributes:

SystemMonitorNotification / Timer / JobChain

JobChain support the following attributes:

SystemMonitorNotification / Timer / Minimum

The following elements must be nested inside a Minimum element:

SystemMonitorNotification / Timer / Maximum

The following elements must be nested inside a Maximum element:

SystemMonitorNotification / Timer / Minimum|Maximum / Script

Script support the following attributes:

Message

Usage

The Message sended to System Monitor can be configured as CDATA element on the following parent nodes:

• SystemMonitorNotification / Notification / NotificationCommand
• SystemMonitorNotification / Notification / NotificationInterface

The Message can contains fixed values and variables. e.g.: <![CDATA[ scheduler id = %MON_N_SCHEDULER_ID%  ]]>

All variables must be defined by using of the %<variable name>% syntax.

The order of the substitution the variables values is:

1. Table variables.
2. Service variables.
3. OS environment variables. 

Tables variables 

 scheduler id = %MON_N_SCHEDULER_ID%, history id = %MON_N_ORDER_HISTORY_ID%, job_chain = %MON_N_JOB_CHAIN_NAME%(%MON_N_ORDER_ID%), error = %MON_N_ERROR_TEXT%

 step from = %MON_SN_STEP_FROM%, step to = %MON_SN_STEP_TO%, notification = %MON_SN_NOTIFICATIONS% (of %MON_SN_MAX_NOTIFICATIONS%)

 timer name = %MON_C_NAME%, text = %MON_C_CHECK_TEXT%

Service variables

 service name = %SERVICE_NAME%

OS environment variables 

All existing system variables can be defined by message with the syntax like %<variable name>% (Windows/Unix).

 %TEMP%/test.exe

Examples 

scheduler id=%MON_N_SCHEDULER_ID%, history id=%MON_N_ORDER_HISTORY_ID%, job_chain=%MON_N_JOB_CHAIN_NAME%(%MON_N_ORDER_ID%), step=%MON_N_ORDER_STEP_STATE%, error=%MON_N_ERROR_TEXT%            

scheduler id=%MON_N_SCHEDULER_ID%, history id=%MON_N_ORDER_HISTORY_ID%, job_chain=%MON_N_JOB_CHAIN_NAME%(%MON_N_ORDER_ID%), steps(%MON_SN_STEP_FROM% to %MON_SN_STEP_TO%), order time elapsed = %MON_N_ORDER_TIME_ELAPSED%s            

name = %MON_C_NAME%, scheduler id=%MON_N_SCHEDULER_ID%, history id=%MON_N_ORDER_HISTORY_ID%, job_chain=%MON_N_JOB_CHAIN_NAME%(%MON_N_ORDER_ID%), steps(%MON_C_STEP_FROM% to %MON_C_STEP_TO%), check = %MON_C_CHECK_TEXT%            

Notification environment variables

The default com.sos.scheduler.notification.plugins.notifier.SystemNotifierProcessBuilderPlugin plugin used by the SystemMonitorNotification / Notification / NotificationCommand element sets the following variables as environment variables:

1. Service variables
2. Tables variables

These variables can be used when the NotificationCommand calls the notification client not directly, but a shell script, that make the logical implementation for sending of the notification messages.

1) configured command in the SystemMonitorNotification_<MonitorSystem>.xml file
<NotificationCommand><![CDATA[/tmp/command.sh]</NotificationCommand>
 
2) content of the /tmp/command.sh file
#! /bin/sh 
# Note: "> /tmp/command_output.txt" used to simulate the starting of the notification client
#
echo $SCHEDULER_MON_SERVICE_NAME:$SCHEDULER_MON_SERVICE_STATUS:$SCHEDULER_MON_SERVICE_MESSAGE_PREFIX history id = $SCHEDULER_MON_TABLE_MON_N_ORDER_HISTORY_ID > /tmp/command_output.txt
 

Service variables

Tables variables

WORK IN PROGRESS

Here you go an example of an XML file used for notifying a specific System Monitor (op5 Monitor) and using NotificationCommand:

<?xml version="1.0" encoding="utf-8"?>
<SystemMonitorNotification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="SystemMonitorNotification_v1.0.xsd">
    <Notification>
    <!--
        configure system monitor service name and command for send notification to OP5 system monitor using nsca Client
        
        %<VARIABLE>% spelling on NotificationCommand element will be used for all Operating Systems (Windows, Unix)
        cmd /c (Windows) e.g /bin/sh -c (Unix) must be separated by | from following command
 
        notification command substitution in this case:
        
        All Environment variables   e.g. %TEMP% or %JAVA_HOME%        
 
        %SERVICE_NAME%              Error Service (service_name_on_error)
 
        %SERVICE_STATUS%            1 if error occured      (service_status_on_error)
                                    0 if error recovered    (service_status_on_success)
 
        %SERVICE_MESSAGE_PREFIX%    ERROR       if error occured
                                    RECOVERED   if error recovered
                                    TIMER       if performance check
 
        %ORDER_HISTORY_ID% ...      table field name of result row for message string(see table definition SCHEDULER_MON_NOTIFICATIONS)   
        -->
        <NotificationMonitor service_name_on_error="Error Service" service_status_on_error="1" service_status_on_success="0">
            <NotificationCommand>
<![CDATA[cmd /c | echo my_nsca_service_host:%SERVICE_NAME%:%SERVICE_STATUS%:%SERVICE_MESSAGE_PREFIX%history id=%ORDER_HISTORY_ID%, step =%ORDER_STEP_STATE%, error=%ERROR_TEXT%, check = %CHECK_TEXT% | C:\nsca\send_nsca.exe -H nsca_server_host -c C:\nsca\send_nsca.cfg -d : ]]>
            </NotificationCommand>
	    </NotificationMonitor>
 
            <NotificationObject>
	    <!-- 
            configure job chains and number of send operations for same problem (if problem is recovered or acknowledged no error notifications will be sended) for sending error notifications 
 
            requirement: monitoring of this job chains must be configured in CheckHistoryConfiguration.xml
            -->
            <JobChains>
                <JobChain notifications="10" name="samples/sample_jobChain_1"/>
                <JobChain notifications="10" name="samples/sample_jobChain_2"/>
            </JobChains>
 
	    <Timers>
	        <Timer>
		<!-- 
                    configure job chains and number of send operations to same check 
 
                    requirement: timer check for this job chain must be configured in CheckHistoryConfiguration.xml
                -->
                    <JobChains>
		        <JobChain notifications="1" name="samples/sample_jobChain_1"/>
		    </JobChains>
         	</Timer>
	    </Timers>
	</NotificationObject>
    </Notification>
</SystemMonitorNotification>

Example SystemMonitorNotification_zabbix.xml

Here is an except of an XML file used for notifying a specific System Monitor (Zabbix Monitor) and using NotificationCommand under Linux

... 
<!--
zabbix_sender 		Zabbix sender installed on the JobScheduler host
localhost 　　　 		Hostname of the zabbix server
Zabbix_server 		JobScheduler Agent name(host name) that registred on Zabbix
samples.job1　　		Item key of zabbix (replace "/" to "." of JOB_NAME
-->
<NotificationCommand>
<![CDATA[/bin/sh -c | zabbix_sender -z localhost -s zabbix_server -k samples.job1 -o %ERROR%]]>
</NotificationCommand>
...

...
<!-- 
Note: "> /tmp/command_output.txt" used to simulate the starting of the notification program
Note: "/bin/sh -c" is separated by "|" from following command
 
%SERVICE_NAME%              value of Attribute "service_name_on_error" or "service_name_on_success"  
%SERVICE_STATUS%            value of Attribute "service_status_on_error" or "service_status_on_success"   
%SERVICE_MESSAGE_PREFIX%    ERROR       if error occured
                            RECOVERED   if error recovered
                            TIMER       if performance check  
%ORDER_HISTORY_ID% ...      table field name of result row for message string(see table definition SCHEDULER_MON_NOTIFICATIONS)
		            e.g.: %ID%, %TASK_ID% etc. 
-->
<NotificationCommand> 
<![CDATA[/bin/sh -c | echo "%JAVA_HOME% %SERVICE_NAME%:%SERVICE_STATUS%:%SERVICE_MESSAGE_PREFIX%history id=%ORDER_HISTORY_ID%, step =%ORDER_STEP_STATE%, error=%ERROR_TEXT%, check = %CHECK_TEXT%" > /tmp/command_output.txt
</NotificationCommand>
...

JobScheduler - Job Chains

Job Chains for these solutions have to be placed under \live\notification. Four Job Chains were implemented for this solution and they have the following functions:

• CheckHistory: reads JobScheduler database tables where the logging is placed, analyses them and writes results into another tables, the Notification tables.
• CleanupNotifications: deletes entries in the Notification tables. Currently this takes place once every day.
• ResetNotifications: sets Status for Notifications in the Notification tables (e.g. Acknowledge)
• SystemNotifier: responsible for notifiying the system Monitor about the current notifications. Moreover, this JobChain is responsible for updating the Notification tables after having notified the System Monitor.

System Monitor

1. The System Monitor receives just passive checks, that means, there are no active checks for monitoring JobScheduler. The only configuration here is the capability to receive passive checks from a remote host.
2. The services in the System Monitor have to be in concordance with the JobScheduler configuration. Passive checks (services) have to be configured and named following the convention used in the XML described above for the JobScheduler (CheckHistoryConfiguration.xml and SystemMonitorNotification_op5.xml).

Use Cases

Recoverable Errors

Initial Situation: A Job Chain is triggered by directory monitoring. That is, when a certain file comes in a monitored folder, the Job Chain starts.

Problem: The Job Chain ended with error.

Handling: The System Monitor will be notified to the service related to the Job Chain with the message error. If a new execution of the Job Chain from a new file end without errors, does not mean that the error is recovered, since the file that has been processed is now another one. That is, the error message at the System Monitor will stay till the same file is again placed in the monitored directory and the Job Chain ends without errors.

Configuration:

• XML CheckConfigurationHistory.xml: Indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor.
• XML SystemMonitorNotification.xml: Specify the name of the Service (in the System Monitor) and specify that it is about a service_name_on_error since you want to have the control when the Job Chain ends in an error.
• System Monitor: Services in the System Monitor have to be configured and named the same way as in the XML file above SystemMonitorNotification.xml.

Workflow Execution takes too long

Initial Situation: A Job Chain is triggered and it could not end, it hanged in a step, taking then longer than expected.

Problem: Execution time was too long

Handling: A timer for this Job Chain is set and the System Monitor will be notified about it. The expiration times for the Job Chains are configured with enough time for processing, that means, this is usually used for cases where the Job Chain hanged in a specific step.

Configuration:

• XML CheckConfigurationHistory.xml: As in the example above, indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor. Moreover, specify the timer for this specific job chain and the function to calculate the expiration time for the timer.
• XML SystemMonitorNotification.xml: As in the example above, specify the name of the Service (in the System Monitor) and specify that it is about a service_name_on_error since you want to have the control when the Job Chain ends in an error. Moreover and essential for this particular case, specify how many times the timer should notify your System Monitor about the expiration of a timer.
• System Monitor: As in the example above, Services in the System Monitor have to be configured and named the same way as in the XML file above SystemMonitorNotification.xml.

SFTP connection refused

Initial Situation: There is a Job Chain that uses SFTP for transferring files. You have a setback configured in this step of the Job Chain, so that if the connection to the SFTP server fails, this step is retried after some time.

Problem: The SFTP server is not available anymore.

Handling: The System Monitor will be notified to the service related to the Job Chain with the message error. However, you don't want to have a bunch of notifications for a Job Chain when is an external factor, the connection to the SFTP Server, what is producing the error.

Configuration:

• XML CheckConfigurationHistory.xml: As in the example above, indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor.
• XML SystemMonitorNotification.xml: As in the example above, specify the name of the Service (in the System Monitor) and specify that it is about a service_name_on_error since you want to have the control when the Job Chain ends in an error. Moreover and very important in this case, specify how many times this Job Chain should notify your System Monitor about the error connecting to the SFTP Server. You can use step_from andstep_to for that in order to reduce the number of notifications for this specific step.
• System Monitor: As in the example above, Services in the System Monitor have to be configured and named the same way as in the XML file above SystemMonitorNotification.xml.

Thresholds

Initial Situation: For example, a specific number of Workflow Executions have to be executed successfully till some specific time. That is, a specific value has to be monitored in order to determine if this quote was reached.

Handling: A new service for History is configured, so that the workflow executions (Job Chains in the JobScheduler vocabulary) send the information that they were executed and finished to the System Monitor.

Configuration:

• XML CheckConfigurationHistory.xml: As in the example above, indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor.
• XML SystemMonitorNotification.xml: Specify the name of the Service (in the System Monitor) but now specify that it is about a service_name_on_success since you want to have the control when the Job Chain ends in an success, and not only when it ends on error.
• System Monitor: As in the example above, Services in the System Monitor have to be configured and named the same way as in the XML file above SystemMonitorNotification.xml.

Acknowledgement

Initial Situation: An alert for a Service has been sent to the System Monitor and a Mail has been sent to the Service Desk (Support Team) notifying about it.

Handling: The problem is well known by the Service Desk and the "acknowledge" the problem. Through the acknowledgement JobScheduler will be notified to and will not send any more notification for this Service to the System Monitor till the Service is again recovered.

Configuration:

• System Monitor: The step of notifying JobScheduler through an acknowledgement in the System Monitor is an execution of a script. This is nothing else than a notification, like sending a mail for instance, but instead, another action is executed, which is the execution of the script that contacts JobScheduler and add an order to the JobChain ResetNotifications described above.