Page History

...

Introduction

This solution is about monitoring JobScheduler and its objects such as Jobs, Job Chains and Orders. Here you get an overview of how JobScheduler monitoring works. This feature will be available starting from general availability release 1.8.

Image Removed

These are some of the features of the architecture of this solution:

• JobScheduler: The architecture establishes a partition between:
  • Detecting errors: A Job Chain analyses the JobScheduler logging and checks whether the monitored Job Scheduler objects had errors or warnings.
  • Sending alerts: Another Job Chain is responsible for sending the alerts to the corresponding System Monitor. The difference here, is that not all alerts are only incidents, but also events, as in occurrences, for example, the alert that a specific Job Chain was executed and which result it ended up with.
• JobScheduler: This architecture allows to analyse the Log History of more than one JobScheduler.
• System Monitor: JobScheduler is able to connect to more than one System Monitor at the same time.

Definitions

• System Monitor: A System Monitor is an instrument to inform the Service Desk (1st Level Support) about incidents in IT systems. It does not serve for the analysis of the incidents, but merely for the information about the incidents, in order to be able to forward and scale these informations.

• Passive Checks: These kind of checks are the ones that are sent remotely from an external host (from the point of view of a System Monitor) to the System Monitor. Otherwise, the ones that are carried out periodically by the System Monitor are called active checks.

• Alerting: An Alert is an alarm, i.e. the message about an event. An alert does not provide every relevant information of an event, but it informs about the existence of the event. An alert can be either positive or negative.

• Notification: The notification of a specific alert. Not every alert will be notified, just the ones that are so configured will be notified. Notifications are therefore a subset of the alerts and can be either positive or negative too.

• Acknowledgement: Is the confirmation of an alert and it has the meaning, that the alert has been seen and/or is well known and the incident is trying to be recovered. An acknowledgement is always manually executed, that means, there is always someone that has realized there is a Critical service and this person acknowledges the services (usually by the Service Desk or 1st Level Support). It is never an automatized step.

Benefits

The benefits of the new solution are:

1. There is no changes to be done in your JobScheduler configuration (Jobs, Job Chains, etc.) in order to get this solution working. You have to add the corresponding Job Chains for the monitoring but do not have to modify your current ones.
2. The whole architecture lies at JobScheduler side and the solution is then independent from the monitor that the alerts are sent to. The solution works for every monitor that can receive passive checks.
3. Processing of Jobs and Job Chains in JobScheduler is not affected or modified by the monitoring, neither in sense of performance nor in sense of stability.
4. The level of detail in a message of a Service in the System Monitor is much higher with this solution. JobScheduler logs very exact what the error is about and this information is sent as a passive check to the specific Service, which shows the log message that JobScheduler logged.
5. The criticality of an error is immediately recognized in the System Monitor. JobScheduler has all information about errors and this information is sorted out and sent to different Services in the System Monitor for every specific case. Through this feature, the Service Desk is immediately able to set its priority for recovering errors. For example, it does not have the same Criticality to recover an error of Performance (low) than when Documents could not be generated (high). Here you go a representation of this feature:

Image Removed

Functionality

• Job Chain and Order Monitoring: Job Chains in JobScheduler can be with the new solution monitored. Actually, the elements that are monitored are the Orders that trigger these Job Chains.

• History Notifications: Not only critical alerts are monitored, but also the positive ones. The history of a specific service is also monitored, to see exactly if a specific workflow was executed or not and what result it ended up with.

• Timers for Job Chains: There are also Timers that measure the performance of a Job Chain. In case it takes too long for a Job Chain to end, a critical alert will be sent to a System Monitor.

• Acknowledgment: Once a service in the System Monitor is critical, there is the possibility to acknowledge this service. That action will add an Order to the JobScheduler, so that JobScheduler does not send more notifications to the System Monitor for this service.

Installation

See https://kb.sos-berlin.com/x/fYEm

Configuration

XML Schemas

XML Schemas and XML files (see examples below) have to be placed together at \config\notification. The schemas are the ones that define which values are allowed in your XML files for the JobScheduler monitoring. That means, you just have to modify your XML files in order to configure the JobScheduler objects you want to monitor and which System Monitor you want to use for that goal, but the XML schemas do not have to be modified.

Schema: SystemMonitorNotification_v1.0.xsd

Description:

1. Specifies delivery way to System Monitor.
2. Specifies notification for error or success conditions
3. Specifies notification for checks of measure the performance of JobScheduler objects

Example SystemMonitorNotification_op5.xml

The configuration element descriptions are organized into the following major categories:

SystemMonitorNotification

SystemMonitorNotification support the following attributes:

<SystemMonitorNotification id="OP5" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="SystemMonitorNotification_v1.0.xsd">

Notification

The following elements may be nested inside a Notification element:

NotificationMonitor

NotificationMonitor support the following attributes:

<!--
NSCA Status:
0 - OK
1 - WARNING
2 - CRITICAL
3 - UNKNOWN
-->
...
<!--
Sending occured errors as CRITICAL
-->

<NotificationMonitor service_name_on_error="Errors">
...
<!--
Sending occured errors as WARNING
-->
<NotificationMonitor service_name_on_error="Errors" service_status_on_error="1">
...        

One of the following elements may be nested inside a NotificationMonitor element:

Here you go an example of an XML file used for notifying a specific System Monitor (op5 Monitor) and using NotificationCommand:

<?xml version="1.0" encoding="utf-8"?>
<SystemMonitorNotification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="SystemMonitorNotification_v1.0.xsd">
    <Notification>
    <!--
        configure system monitor service name and command for send notification to OP5 system monitor using nsca Client
        
        %<VARIABLE>% spelling on NotificationCommand element will be used for all Operating Systems (Windows, Unix)
        cmd /c (Windows) e.g /bin/sh -c (Unix) must be separated by | from following command
 
        notification command substitution in this case:
        
        All Environment variables   e.g. %TEMP% or %JAVA_HOME%        
 
        %SERVICE_NAME%              Error Service (service_name_on_error)
 
        %SERVICE_STATUS%            1 if error occured      (service_status_on_error)
                                    0 if error recovered    (service_status_on_success)
 
        %SERVICE_MESSAGE_PREFIX%    ERROR       if error occured
                                    RECOVERED   if error recovered
                                    TIMER       if performance check
 
        %ORDER_HISTORY_ID% ...      table field name of result row for message string(see table definition SCHEDULER_MON_NOTIFICATIONS)   
        -->
        <NotificationMonitor service_name_on_error="Error Service" service_status_on_error="1" service_status_on_success="0">
            <NotificationCommand>
<![CDATA[cmd /c | echo my_nsca_service_host:%SERVICE_NAME%:%SERVICE_STATUS%:%SERVICE_MESSAGE_PREFIX%history id=%ORDER_HISTORY_ID%, step =%ORDER_STEP_STATE%, error=%ERROR_TEXT%, check = %CHECK_TEXT% | C:\nsca\send_nsca.exe -H nsca_server_host -c C:\nsca\send_nsca.cfg -d : ]]>
            </NotificationCommand>
	    </NotificationMonitor>
 
            <NotificationObject>
	    <!-- 
            configure job chains and number of send operations for same problem (if problem is recovered or acknowledged no error notifications will be sended) for sending error notifications 
 
            requirement: monitoring of this job chains must be configured in CheckHistoryConfiguration.xml
            -->
            <JobChains>
                <JobChain notifications="10" name="samples/sample_jobChain_1"/>
                <JobChain notifications="10" name="samples/sample_jobChain_2"/>
            </JobChains>
 
	    <Timers>
	        <Timer>
		<!-- 
                    configure job chains and number of send operations to same check 
 
                    requirement: timer check for this job chain must be configured in CheckHistoryConfiguration.xml
                -->
                    <JobChains>
		        <JobChain notifications="1" name="samples/sample_jobChain_1"/>
		    </JobChains>
         	</Timer>
	    </Timers>
	</NotificationObject>
    </Notification>
</SystemMonitorNotification>

Example SystemMonitorNotification_zabbix.xml

Here is an except of an XML file used for notifying a specific System Monitor (Zabbix Monitor) and using NotificationCommand under Linux

... 
<!--
zabbix_sender 		Zabbix sender installed on the JobScheduler host
localhost 　　　 		Hostname of the zabbix server
Zabbix_server 		JobScheduler Agent name(host name) that registred on Zabbix
samples.job1　　		Item key of zabbix (replace "/" to "." of JOB_NAME
-->
<NotificationCommand>
<![CDATA[/bin/sh -c | zabbix_sender -z localhost -s zabbix_server -k samples.job1 -o %ERROR%]]>
</NotificationCommand>
...

...
<!-- 
Note: "> /tmp/command_output.txt" used to simulate the starting of the notification program
Note: "/bin/sh -c" is separated by "|" from following command
 
%SERVICE_NAME%              value of Attribute "service_name_on_error" or "service_name_on_success"  
%SERVICE_STATUS%            value of Attribute "service_status_on_error" or "service_status_on_success"   
%SERVICE_MESSAGE_PREFIX%    ERROR       if error occured
                            RECOVERED   if error recovered
                            TIMER       if performance check  
%ORDER_HISTORY_ID% ...      table field name of result row for message string(see table definition SCHEDULER_MON_NOTIFICATIONS)
		            e.g.: %ID%, %TASK_ID% etc. 
-->
<NotificationCommand> 
<![CDATA[/bin/sh -c | echo "%JAVA_HOME% %SERVICE_NAME%:%SERVICE_STATUS%:%SERVICE_MESSAGE_PREFIX%history id=%ORDER_HISTORY_ID%, step =%ORDER_STEP_STATE%, error=%ERROR_TEXT%, check = %CHECK_TEXT%" > /tmp/command_output.txt
</NotificationCommand>
...

...
<!-- 
Note: "/bin/sh -c" is separated by "|" from following command
-->
<NotificationCommand><![CDATA[/bin/sh -c | /tmp/command.sh]</NotificationCommand>
...
 
 

article describes individual configuration parameters and provides examples of their use with monitors such as op5 and Zabbix and using of the mail und JMS interfaces.

Send notifications

Notify on error

SystemMonitorNotification / Notification / NotificationMonitor / @service_name_on_error is configured.

SystemMonitorNotification / Notification / NotificationObjects / JobChain

Error messages

• will be sent:
  • when at the time of the sos / notification / SystemNotifier run, an order is in a job chain state(step) that has ended with an error
• will not be sent:
  • when after the last run of the sos / notification / SystemNotifier, an error has occured in an job chain state(step) but at the time of the current sos / notification / SystemNotifier run this order is in an other/next job chain state(step)
    • this kind of error is ignored because the order has continued to run
  • when an error has reoccurred in the same job chain state(step) where a notification has already been sent
    • this order state is considered as notified and no new notification will be sent
      • e.g. an job chain state(step) has been restarted manually or by a setback
    • this behaviour has been changed with 

      Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JITL-534

      - providing support for repeatedly failed executions.
  • when the first step of the specific order has been removed from the notification tables by sos / notification / CleanupNotifications
    • this behaviour has been changed with 

      Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JITL-516

      - providing support for long running orders
      
  • when a notification maximum has been reached
    • see  SystemMonitorNotification / Notification / NotificationObjects / JobChain / @notifications
  • when a job chain state(step) has been configured as excluded
    • see SystemMonitorNotification / Notification / NotificationObjects / JobChain / @excluded_steps
  • when the @step_from or @step_to settings have been configured and the job chain state(step) is out of the configured range
    • see SystemMonitorNotification / Notification / NotificationObjects / JobChain / @step_from | @step_to

  • when the @return_code_from or @return_code_to settings have been configured and the return code of the job chain state(step) is out of the configured range
    • see SystemMonitorNotification / Notification / NotificationObjects / JobChain / @return_code_from | @return_code_to


Recovery messages

• will be automatically sent using the same service name and message as the relevant error message:
  • when the error message of a job chain state(step) has already been sent and the order at the time of the current sos / notification / SystemNotifier run is in an other/next state(step)
    • e.g. the rerun of the error state(step) has been successfull and the order has been moved to the next job chain state(step)
    • Note: use ${SERVICE_STATUS} and ${SERVICE_MESSAGE_PREFIX} variables to differentiate between recovery and error message
      • See:
        • Message / Variables / Service variables
        • Notification environment variables / Service variables
        • Examples / Examples System Monitoring / NotificationCommand ( Nagios / OP5 )
        • Examples / Examples System Monitoring / NotificationCommand ( Nagios / Opsview )
• will not be sent:
  • when a job chain state(step) has recovered after the last run of the sos / notification / SystemNotifier but at the time of the current sos / notification / SystemNotifier run a new error in the other/next step has occured

SystemMonitorNotification / Notification / NotificationObjects / Job

Error messages

• will be sent:
  • when a job chain state(step) or standalone job (JobScheduler versions from 1.12) ends with an error
• will not be sent:
  • when a notification maximum has been reached
    • see SystemMonitorNotification / Notification / NotificationObjects / Job / @notifications
  • when the @return_code_from or @return_code_to settings have been configured and the return code of the job is out of the configured range
    • see SystemMonitorNotification / Notification / NotificationObjects / Job / @return_code_from | @return_code_to


Notify on success

SystemMonitorNotification / Notification / NotificationMonitor / @service_name_on_success is configured.

SystemMonitorNotification / Notification / NotificationObjects / JobChain

Success messages

• will be sent:
  • when an order is completed and the last job chain state(step) has no error
• will not be sent:
  • when a notification maximum has been reached
    • see SystemMonitorNotification / Notification / NotificationObjects / JobChain / @notifications
  • when the @return_code_from or @return_code_to settings have been configured and the return code of the job chain state(step) is out of the configured range

    • see SystemMonitorNotification / Notification / NotificationObjects / JobChain / @return_code_from | @return_code_to


SystemMonitorNotification / Notification / NotificationObjects / Job

Success messages

• will be sent:
  • when a job chain state(step) or standalone job (JobScheduler releases from 1.12) ends without error
• will not be sent:
  • when a notification maximum has been reached
    • see SystemMonitorNotification / Notification / NotificationObjects / Job / @notifications
  • when the @return_code_from or @return_code_to settings have been configured and the return code of the job is out of the configured range
    • see SystemMonitorNotification / Notification / NotificationObjects / Job / @return_code_from | @return_code_to


Configuration Editor

We recommend that the XML Editor is used generate monitoring configuration objects. This editor automatically uses an XSD Schema to generate configuration suggestions and validate configurations, and its use is intended to provide a significant reduction in the time required to develop and test a configuration.

XSD Schema locations

• https://www.sos-berlin.com/schema/jobscheduler/SystemMonitorNotification_v1.0.xsd
• JobScheduler releases before 1.13.1
  • <scheduler_data>/config/notification/SystemMonitorNotification_v1.0.xsd
• JobScheduler releases starting from 1.13.1
  • <scheduler_data>/config/live/sos/.configuration/notification/SystemMonitorNotification_v1.0.xsd

Configuration

JobScheduler

Activation of Monitoring Interface 

• JobScheduler releases before 1.11
  
  • see JobScheduler - Job Chains
• JobScheduler releases starting from 1.11
  
  • Set param sos.use_notification true (config/scheduler.xml)
  • see JobScheduler - Job Chains

Note: 

• JobScheduler releases before 1.13.1
  • SystemMonitorNotification_<MonitorSystem>.xml file(s) (see below) must be configured before activtion.
    
• JobScheduler releases starting from 1.13.1
  • SystemMonitorNotification_<MonitorSystem>.xml file(s) (see below) must be configured before activtion or
    
  • The NOTIFICATION configuration was forwarded by the Joc Cockpit to the respective JobScheduler Master.

SystemMonitorNotification files

JobScheduler releases before 1.13.1

Location: <scheduler_data>/config/notification

JobScheduler releases starting from 1.13.1

Note: Usage of the configuration files in the <scheduler_data>/config/notification folder has been deprecated.

Location:  <scheduler_data>/config/live/sos/.configuration/notification

SystemMonitorNotification Elements

The configuration element descriptions are organized into the following major categories:

SystemMonitorNotification

SystemMonitorNotification supports the following attributes:

<SystemMonitorNotification system_id="op5">
...

SystemMonitorNotification / Notification

Notification supports the following attributes:

...
<Notification name="Mail: on failed job">
...

The following elements may be nested inside a Notification element:

SystemMonitorNotification / Notification / NotificationMonitor

The JobScheduler Interface Monitor can be used to monitor the messages for the 3 use cases:

• error case
  • an error has occurred / been recovered during a job chain / job execution
    • the service_name_on_error setting is responsible for this monitoring case
• success case
  • a job chain / job ends successfully
    • the service_name_on_success setting is responsible for this monitoring case
• performance check (see Timer)
  • usually the service_name_on_error setting is responsible for this monitoring case but the performance check will also work if only the service_name_on_success setting has been defined.

In addition, the service_name_on_error / service_name_on_success attributes have the following meaning:

• NotificationInterface
  • The setting must match the corresponding service name in the System Monitor such as Nagios or op5.
• NotificationCommand
  • Freely selectable, has no further meaning than to identify a notification.
• NotificationMail
  • Freely selectable, has no further meaning than to identify a notification.
• NotificationJMS
  • The setting must match the corresponding queue/topic name in the JMS Server.

Note:

• attributes service_name_on_error and service_name_on_success
  • at least one of these attributes must be configured
  • both attributes can be configured together
  •  the use of this settings must be unique within one SystemNotification
    
    

NotificationMonitor supports the following attributes:

<!-- Example 
op5 NSCA Status: 
0 - OK 
1 - WARNING 
2 - CRITICAL 
3 - UNKNOWN --> 
... 
<!-- Sending errors as CRITICAL (default) --> 
<NotificationMonitor service_name_on_error="JobScheduler Monitoring Errors"> 
... 
<!-- Sending occurred errors as WARNING --> 
<NotificationMonitor service_name_on_error="JobScheduler Monitoring Errors" service_status_on_error="1"> 
...

One of the following elements must be nested inside a NotificationMonitor element:

SystemMonitorNotification / Notification / NotificationMonitor / NotificationInterface

NSCA plug-in Interface to be executed for System Monitor notification.

NotificationInterface supports the following attributes:

See : Message, Examples

...
<NotificationInterface monitor_host="monitor_host" monitor_port="5667" monitor_encryption="XOR" service_host="service_host"><![CDATA[
scheduler id=${MON_N_SCHEDULER_ID}, history id=${MON_N_ORDER_HISTORY_ID}, job_chain=${MON_N_JOB_CHAIN_NAME}(${MON_N_ORDER_ID}), step =${MON_N_ORDER_STEP_STATE}, error=${MON_N_ERROR_TEXT}
]]></NotificationInterface>
...

SystemMonitorNotification / Notification / NotificationMonitor / NotificationCommand

Command line to be executed for System Monitor notification.

NotificationCommand supports the following attributes:

See : Message, Examples

...
<NotificationCommand><![CDATA[
echo scheduler id=${MON_N_SCHEDULER_ID}, history id=${MON_N_ORDER_HISTORY_ID}, job_chain=${MON_N_JOB_CHAIN_NAME}(${MON_N_ORDER_ID}), step =${MON_N_ORDER_STEP_STATE}, error=${MON_N_ERROR_TEXT} > D://errors.txt
]]></NotificationCommand>
...

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail

The Mail interface to be executed for System Monitor notification.

The Mail interface reads the following values from the configuration files:

• config/factory.ini
  
  • Section spooler
    • log_mail_from
    • log_mail_to
    • log_mail_cc
    • log_mail_bcc
    • smtp
    • mail_queue_dir
    • mail_queue_only
  • Section smtp
    • mail.smtp.user
    • mail.smtp.password
    • mail.smtp.port
    • mail.smtp.connectiontimeout
    • mail.smtp.timeout
• config/private/private.conf
  • joc.url
    

NotificationMail supports the following attributes:

The following elements can be nested inside a NotificationMail element:

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail / From

E-mail address of the account that sends the e-mail.

The mail notification interface uses the value of the log_mail_from entry (configuration file config/factory.ini) when this element is not set.

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail / To

E-mail address of the recipient(s) of a notification e-mail.

When this element

• is not set
  • log_mail_to will be used
• is set
  •   log_mail_to, log_mail_cc, log_mail_bcc are not used
    

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail / CC

E-mail address of the recipient(s) of a carbon copy notification e-mail.

When this element

• is not set
  • log_mail_cc will be used (if the NotificationMail/To element is not defined - see above)
    
• is set
  •   log_mail_cc, log_mail_bcc are not used

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail / BCC

E-mail address of recipient(s) of a blind carbon copy notification e-mail.

When this element

• is not set
  • log_mail_bcc will be used (if the NotificationMail/To or NotificationMail/CC elements are not defined - see above)
    
• is set
  •   log_mail_bcc are not used

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail / Subject

Subject of an e-mail notification.

The Subject can contain the JobScheduler Monitoring Interface variables.

See : Message, Examples

...
<Subject><![CDATA[JobScheduler notification: ${SERVICE_MESSAGE_PREFIX}, job executed with errors: ${MON_N_JOB_NAME}]]></Subject>
...

SystemMonitorNotification / Notification / NotificationMonitor / NotificationMail / Body

Body of an e-mail notification.

The Body can contain the JobScheduler Monitoring Interface variables.

See : Message, Examples

<!DOCTYPE html>
<html lang="en">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
    <style>.tg  {border-collapse:collapse;border-spacing:0;border-color:#aaa;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:#aaa;color:#333;background-color:#fff;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:bold;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:#aaa;color:#fff;background-color:#f38630;}</style>
    <title>JobScheduler Notification</title>
</head>
<body>
<table class="tg">
    <tr>
        <th colspan="4">Error</th>
    </tr>
    <tr>
        <td>Code:</td><td>${MON_N_ERROR_CODE}</td>
        <td>Messsage</td><td>${MON_N_ERROR_TEXT}</td>
    </tr>
    <tr>
        <th colspan="4">JobScheduler</th>
    </tr>
    <tr>
        <td>JobScheduler ID</td><td>${MON_N_SCHEDULER_ID}</td>
        <td>Agent URL</td><td>${MON_N_AGENT_URL}</td>
    </tr>
    <tr>
        <th colspan="4">Order</th>
    </tr>
    <tr>
        <td>Order ID</td><td><a href="${JOC_HREF_ORDER}">${MON_N_ORDER_ID}</a></td>
        <td>Order Title</td><td>${MON_N_ORDER_TITLE}</td>
    </tr>
    <tr>
        <td>Job Chain Name</td><td><a href="${JOC_HREF_JOB_CHAIN}">${MON_N_JOB_CHAIN_NAME}</a></td>
        <td>Job Chain Title</td><td>${MON_N_JOB_CHAIN_TITLE}</td>
    </tr>
    <tr>
        <td>Job Name</td><td><a href="${JOC_HREF_JOB}">${MON_N_JOB_NAME}${MON_N_JOB_NAME}</a></td>
        <td>Job Title</td><td>${MON_N_JOB_TITLE}</td>
    </tr>
    <tr>
        <th colspan="4">Task History</th>
    </tr>
    <tr>
        <td>Task ID</td><td>${MON_N_TASK_ID}</td>
        <td>Time elapsed</td><td>${MON_N_TASK_TIME_ELAPSED}</td>
    </tr>
    <tr>
        <td>Start Time</td><td>${MON_N_TASK_START_TIME}</td>
        <td>End Time</td><td>${MON_N_TASK_END_TIME}</td>
    </tr>
</table>
</body>
</html>

SystemMonitorNotification / Notification / NotificationMonitor / NotificationJMS

JMS Interface to be executed for System Monitor notification.

Note: the provider-specific queue or topic name will be defined with the service_name_on_error / service_status_on_succes attribute of the parent SystemMonitorNotification / Notification / NotificationMonitor element.

NotificationJMS supports the following attributes: 

One of the following elements must be nested inside a NotificationJMS element:

JMS message:

SystemMonitorNotification / Notification / NotificationMonitor / NotificationJMS / ConnectionFactory

Specifies use of a JMS ConnectionFactory implementation.

ConnectionFactory supports the following attributes:  

 The following element can be nested inside a ConnectionFactory element:

SystemMonitorNotification / Notification / NotificationMonitor / NotificationJMS / ConnectionFactory / ConstructorArguments

The following elements can be nested inside a ConstructorArguments element:

...
<ConnectionFactory java_class="org.apache.activemq.ActiveMQConnectionFactory">
    <ConstructorArguments>
        <Argument type="java.lang.String"><![CDATA[tcp://localhost:61616]]></Argument>
    </ConstructorArguments>
</ConnectionFactory>
...

...
<ConnectionFactory java_class="org.apache.activemq.ActiveMQConnectionFactory">
    <ConstructorArguments>
        <Argument type="java.lang.String"><![CDATA[my_user_name]]></Argument>
        <Argument type="java.lang.String"><![CDATA[my_password]]></Argument>
        <Argument type="java.lang.String"><![CDATA[tcp://localhost:61616]]></Argument>
    </ConstructorArguments>
</ConnectionFactory>
...

SystemMonitorNotification / Notification / NotificationMonitor / NotificationJMS / ConnectionFactory / ConstructorArguments / Argument

 Argument supports the following attributes: 

The value of the constructor argument will be stored as CDATA node.

...
<Argument type="java.lang.String"><![CDATA[tcp://localhost:61616]]></Argument>
...

SystemMonitorNotification / Notification / NotificationMonitor / NotificationJMS / ConnectionJNDI

Specifies use of a JNDI properties file to create a JNDI IntialContextFactory.

See: Connecting to the JMS Server by Using JNDI

ConnectionJNDI supports the following attributes:  

java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory
java.naming.provider.url=tcp://localhost:61616 

SystemMonitorNotification / Notification / NotificationMonitor / NotificationJMS / Message

Body of a JMS notification.

See : Message, Examples

SystemMonitorNotification / Notification / NotificationObjects

One of the following elements must be nested inside a NotificationObjects element:

<SystemMonitorNotification system_id="op5"> 
  <Notification> 
    <NotificationMonitor service_name_on_error="JobScheduler Monitoring Errors"> 
      ... 
    </NotificationMonitor> 
    <NotificationObjects>
      <!-- Send the job error, occurrent in the "test/my_job" order job, to the "JobScheduler Monitoring Errors" service. -->        
      <Job name="test/my_job" /> 
      <!-- Send the job chain error, occurrent in the "test/my_jobchain" job chain, to the "JobScheduler Monitoring Errors" service. --> 
      <JobChain name="test/my_jobchain" /> 
    </NotificationObjects> 
 </Notification> 
</SystemMonitorNotification>  

SystemMonitorNotification / Notification / NotificationObjects / Job 

This element specifies the order-controlled or standalone jobs for which notifications are being sent to a system monitor.

Support for standalone jobs starting with

Job supports the following attributes:

...
<Job notifications="2" name="test/my_job"/>
...
<Job scheduler_id="scheduler_4444" />
...
<Job scheduler_id="scheduler_4444" name="test/my_.*" />
... 
<Job name="test/my_job" return_code_from="5"/> 
...  
<Job name="test/my_job" return_code_to="10"/>
...  
<Job name="test/my_job" return_code_from="5" return_code_to="5"/>  
...

SystemMonitorNotification / Notification / NotificationObjects / JobChain

Specifies the job chains for which notifications are being sent to a system monitor.
The element can be repeatedly used to specifiy a number of job chains.

Default behaviour for repeatedly failed job chain steps: when an error reoccurrs in the same job node for which a notification has already been sent then this order state is considered being previously notified and no new notification will be sent.
See child element NotifyRepeatedError.

JobChain supports the following attributes:

The following element can be nested inside a JobChain element:

...
<JobChain notifications="2" name="test/my_jobchain"/>
...
<JobChain scheduler_id="scheduler_4444" />
...
<JobChain scheduler_id="scheduler_4444" name="test/my_.*" />
... 
<JobChain name="test/my_jobchain" return_code_from="5"/> 
... 
<JobChain name="test/my_jobchain" return_code_to="10"/>
... 
<JobChain name="test/my_jobchain" return_code_from="5" return_code_to="5"/>  
...
<JobChain name="test/my_jobchain" step_from="200"/>
...
<JobChain name="test/my_jobchain" step_to="500"/>
...
<JobChain name="test/my_jobchain" step_from="300" step_to="300"/>
...
<JobChain name="test/my_jobchain" excluded_steps="200;300"/>
...
<JobChain name="test/my_jobchain">
  <NotifyRepeatedError />
</JobChain>
...

SystemMonitorNotification / Notification / NotificationObjects / JobChain / NotifyRepeatedError

Send notifications for all errors that occur, do not suppress errors for repeatedly failed executions.

One of the following elements can be nested inside a NotifyRepeatedError element:

...
<JobChain name="test/my_jobchain">
  <NotifyRepeatedError>
      <NotifyByIntervention />
  </NotifyRepeatedError>
</JobChain>
...
<JobChain name="test/my_jobchain">
  <NotifyRepeatedError>
      <NotifyByPeriod period="5h 30m" />
  </NotifyRepeatedError>
</JobChain>
...
<JobChain name="test/my_jobchain">
  <NotifyRepeatedError>
      <NotifyByIntervention />
      <NotifyByPeriod period="2h" />
  </NotifyRepeatedError>
</JobChain>
...

SystemMonitorNotification / Notification / NotificationObjects / JobChain / NotifyRepeatedError / NotifyByIntervention

Send notifications for errors that occur due to repeated failed executions if the restart was caused by manual intervention.

SystemMonitorNotification / Notification / NotificationObjects / JobChain / NotifyRepeatedError / NotifyByPeriod

Send notifications for errors that occur due to repeatedly failed executions if a configurable period of time is exceeded.

NotifyByPeriod supports the following attributes:

SystemMonitorNotification / Notification / NotificationObjects / TimerRef

TimerRef supports the following attributes:

<SystemMonitorNotification system_id="op5"> 
  <Notification> 
    <NotificationMonitor service_name_on_error="JobScheduler Monitoring Error"> 
      ... 
    </NotificationMonitor> 
    <NotificationObjects> 
     <!-- 
     Send the job chain error, occurring in the "test/my_jobchain" job chain, to the "JobScheduler Monitoring Errors" service. 
     --> 
     <JobChain name="test/my_jobchain" /> 
    </NotificationObjects> 
  </Notification>   
 
  <Notification> 
    <NotificationMonitor service_name_on_error="JobScheduler Monitoring Performance"> 
      ... 
    </NotificationMonitor> 
    <NotificationObjects> 
      <!-- 
      Sends the performance check error, occurring in the "test/my_jobchain" job chain, to the "JobScheduler Monitoring Performance" service. 
      Sends the performance check error to the "JobScheduler Monitoring Performance" service will be ignored when the "test/my_jobchain" has the job chain error (default notify_on_error = false). 
      --> 
      <TimerRef ref="my_timer" /> 
    </NotificationObjects> 
 </Notification>   
 
 <Timer name="my_timer"> 
    <TimerJobChain name="test/my_jobchain" /> 
 </Timer> 
</SystemMonitorNotification> 

SystemMonitorNotification / Notification / NotificationObjects / MasterMessage

MasterMessage includes problems detected by a JobScheduler Master, e.g. database connection lost.

Requirements:

•   ./config/factory.ini configuration file:

  • mail_queue_only=true

  • mail_queue_dir setting specifies a directory to store the JobScheduler mails

  • mail_on_warning=true

  • mail_on_error=true

MasterMessage supports the following attributes:

SystemMonitorNotification / Notification / NotificationObjects / TaskWarning

TaskWarning includes job execution warning messages.

Requirements:

•  ./config/factory.ini configuration file:

  • mail_queue_only=true

  • mail_queue_dir setting specifies a directory to store the JobScheduler mails

  • mail_on_warning=true

TaskWarning supports the following attributes:

SystemMonitorNotification / Notification / NotificationObjects / TaskIfLongerThan

TaskIfLongerThan includes the feature to send notification in case that the execution of job requires a longer duration than expected.

Requirements:

• The Job configuration contains the warn_if_longer_than setting.  
• ./config/factory.ini configuration file:

  • mail_queue_only=true

  • mail_queue_dir setting specifies a directory to store the JobScheduler mails

  • mail_on_warning=true

TaskIfLongerThan supports the following attributes:

SystemMonitorNotification / Notification / NotificationObjects / TaskIfShorterThan

 TaskIfShorterThan includes the feature to send notification in case that the execution of job requires a shorter duration than expected.

 Requirements:

• The Job configuration contains the warn_if_shorter_than setting.  

• ./config/factory.ini configuration file:

  • mail_queue_only=true

  • mail_queue_dir setting specifies a directory to store the JobScheduler mails

  • mail_on_warning=true

 TaskIfShorterThan supports the following attributes:

SystemMonitorNotification / Timer 

The following elements must be nested inside a Timer element:

<SystemMonitorNotification system_id="op5"> 
  ... 
  <Timer name="my_timer_1"> 
    <TimerJobChain name="test/my_jobchain_1" />
    <TimerJob name="test/my_job_1" /> 
    <Maximum><Script language="javascript"><![CDATA[1000]]></Script></Maximum> 
  </Timer> 
 
  <Timer name="my_timer_2"> 
    <TimerJobChain name="test/my_jobchain_2" /> 
    <TimerJobChain name="test/my_jobchain_3" /> 
    <Minimum><Script language="javascript"><![CDATA[500]]></Script></Minimum> 
    <Maximum><Script language="javascript"><![CDATA[1000]]></Script></Maximum> 
  </Timer> 
</SystemMonitorNotification> 

Timer supports the following attributes:

...
<Timer name="my_timer">
... 

SystemMonitorNotification / Timer / TimerJob

TimerJob supports the following attributes: 

...
<TimerJob scheduler_id="scheduler_4444" /> 
... 
<TimerJob scheduler_id="scheduler_4444" name="test/my_.*" /> 
... 
<TimerJob name="test/my_job"/> 
... 

SystemMonitorNotification / Timer / TimerJobChain

TimerJobChain supports the following attributes:

...
<TimerJobChain scheduler_id="scheduler_4444" /> 
... 
<TimerJobChain scheduler_id="scheduler_4444" name="test/my_.*" /> 
... 
<TimerJobChain name="test/my_jobchain" step_from="200"/> 
... 
<TimerJobChain name="test/my_jobchain" step_to="500"/> 
... 
<TimerJobChain name="test/my_jobchain" step_from="300" step_to="300"/>
...

SystemMonitorNotification / Timer / Minimum

The following elements must be nested inside a Minimum element:

...
<Timer name="my_timer">
  ...
  <Minimum><Script language="javascript"><![CDATA[1000]]></Script></Minimum>
</Timer>
... 

SystemMonitorNotification / Timer / Maximum

The following elements must be nested inside a Maximum element:

...
<Timer name="my_timer">
  ...
  <Maximum><Script language="javascript"><![CDATA[1000]]></Script></Maximum>
</Timer>
... 

SystemMonitorNotification / Timer / Minimum|Maximum / Script

Script supports the following attributes:

 The Script element can contain:

• a fixed value
• a calculation based on the job/order parameters

Fixed value

A fixed value is the time allowed in seconds for the specific Minimum or Maximum definition

...
  <Script language="javascript"><![CDATA[1000]]></Script>
...

Calculation

The calculation is to result in the time in seconds for the specific Minimum or Maximum definition.

This example calculates the execution time depending on the ${file_size} parameter that was set by a specific job (see the example below)´.

...
  <Script language="javascript"><![CDATA[                     
    function my_calculate(){                         
      var fileSize              = new java.lang.Double(${file_size});                         
      var timerExpiryFactor     = 0.0025;                         
      var timerExpiryTolerance  = timerExpiryFactor*0.1;                         
      var timerExpiry           = new java.lang.Double(timerExpiryFactor+timerExpiryTolerance);                         
      timerExpiry               = timerExpiry*fileSize;                     
      return timerExpiry;                     
    }                         
    my_calculate();
  ]]></Script>
...

This example job calculates and creates a new order parameter file_size.

To store the parameters into database (table SCHEDULER_MON_RESULTS) :

• set the scheduler_notification_result_parameters parameter (see job documentation jobs/JobSchedulerNotificationStoreResultsJob.xml)
• set the StoreResultsJobJSAdapterClass as monitor
  • JobScheduler releases before 1.11
    • com.sos.scheduler.notification.jobs.result.StoreResultsJobJSAdapterClass

  • JobScheduler releases starting from 1.11
    

    • com.sos.jitl.notification.jobs.result.StoreResultsJobJSAdapterClass

<?xml version="1.0" encoding="ISO-8859-1"?> 
<job  title="Sample Job with Store Result Monitor" order="yes" stop_on_error="no" tasks="1">     
  <params>
     <!--
     set the scheduler_notification_result_parameters parameter
     -->         
    <param name="scheduler_notification_result_parameters" value="file_size"/>     
  </params>     
  
  <!--
  calculate and create the new order parameter if necessary
  -->
  <script language="java:javascript"><![CDATA[             
      function spooler_process(){                                  
        var order    = spooler_task.order;                 
        var params   = spooler.create_variable_set();                 
        params.merge(spooler_task.params);                 
        params.merge(order.params);                      
        
        // parameter scheduler_file_path was set in the previous job chain step
        var file     = new java.io.File(params.value("scheduler_file_path"));                 
        var fileSize = file.length()/1024;                 
        order.params.set_var("file_size",fileSize.toString());                          
      return true;             
      }]]>     
   </script>          
 
   <!-- 
   set the StoreResultsJobJSAdapterClass as a monitor
   -->     
   <monitor  name="notification_monitor" ordering="1">         
     	<!-- JobScheduler releases before 1.11
	<script java_class="com.sos.scheduler.notification.jobs.result.StoreResultsJobJSAdapterClass" language="java"/>
        -->
   
        <!-- JobScheduler releases starting from 1.11 --> 
        <script java_class="com.sos.jitl.notification.jobs.result.StoreResultsJobJSAdapterClass" language="java"/> 
   </monitor>

   <run_time /> 
</job> 

Message

Anchor
message message

Usage

The Message can be configured on the following parent nodes as a CDATA element :

• SystemMonitorNotification / Notification / NotificationInterface

• SystemMonitorNotification / Notification / NotificationCommand
• SystemMonitorNotification / Notification / NotificationMail 
  • Subject

  • Body
• SystemMonitorNotification / Notification / NotificationJMS / Message


The Message can contain: 

• Fixed values
• Variables

Example: <![CDATA[ scheduler id = ${MON_N_SCHEDULER_ID}  ]]>

Variables

All variables (except OS environment variables) must be defined by using of the

${<variable name>}

syntax.

Note:

• Syntax for the JobScheduler version 1.10.6 and higher. Syntax for the JobScheduler version 1.10.4, 1.10.5 (see below) is still supported.
• Syntax for the JobScheduler version 1.10.4, 1.10.5:  {<variable name>}
• Syntax for the JobScheduler previous versions: %<variable name>%

The order of the substitution the variables values is:

1. Table variables.
2. Service variables.
3. JOC Cockpit variables.
4. OS environment variables.
   
   

Table variables

${MON_N_ID}

 scheduler id = ${MON_N_SCHEDULER_ID}, history id = ${MON_N_ORDER_HISTORY_ID}, job_chain = ${MON_N_JOB_CHAIN_NAME}(${MON_N_ORDER_ID}), error = ${MON_N_ERROR_TEXT}

${MON_SN_OBJECT_TYPE}

${MON_SN_OBJECT_TYPE_NAME}

${MON_SN_TITLE}

${MON_SN_RETURN_CODE_FROM} 

${MON_SN_RETURN_CODE_TO}

 step from = ${MON_SN_STEP_FROM}, step to = ${MON_SN_STEP_TO}, notification = ${MON_SN_CURRENT_NOTIFICATION} (of ${MON_SN_NOTIFICATIONS})