What logging possibilities does JobScheduler provide?

Introduction

• The JobScheduler creates a number of logs to provide specific information about jobs, job chains, orders, tasks and JobScheduler operation.
• Log files can be accessed on the file system using a text editor: file locations are included in the current article.
• Some JobScheduler Master and Agent Log files can also be accessed by users of the JOC Cockpit Operating Interface. Note that users have to have been granted the necessary permissions before they can access log files. In addition, the JobScheduler Web Services provide extensive logging. See the JOC Cockpit - Logging article for more information.
  
  • Note also that the JOC Cockpit also provides access to an Audit Log of all interventions carried out over the JOC Cockpit that modify the status of Jobs, Job Chains and Orders. See the JOC Cockpit - Audit Log article for more information.

Log File Location

• The location of the the log files is specified by the log_dir parameter in the ${SCHEDULER_DATA}/config/factory.ini file where ${SCHEDULER_DATA} is the JobScheduler data directory that has been specified during installation. The default location is the ${SCHEDULER_DATA}/logs folder. 

Custom Log File Location

• Administrators are free to specify an alternative location for the JobScheduler log files. In this situation, to ensure that the JOC Cockpit is able to find the alternative log file location, we recommend that administrators do not change the log_dir parameter in the factory.ini file but create a SymLink pointing to the new log directory in place of the default folder specified by the log_dir parameter. The JOC cockpit will then look at the default log file location and follow the SymLink to the new location.
• To Configure a Custom Log File location:
  • Stop the JobScheduler Master (to allow the original logs folder to be moved/deleted/renamed)
  • Move/delete/rename the original logs folder
  • On Window: Open a command prompt window (with elevated Administrator rights)
    
    • Navigate to the current location of the logs folder
    • Enter: mklink /D logs Target
      • where Target is the new name/location (absolute or relative) of the logs directory
  • On Linux: Open a shell
    
    • Navigate to the current location of the logs folder
    • Enter: ln -s "Target" logs
      • where Target is the new name/location (absolute or relative) of the logs directory
  • Start the JobScheduler Master

JobScheduler Log Files

JobScheduler Master Log Files

JobScheduler Agent Log Files

Feature Availability - features introduced with release 1.10

JS-1329 - Getting issue details... STATUS

JS-1393 - Getting issue details... STATUS

JS-1497 - Getting issue details... STATUS

JITL-95 - Getting issue details... STATUS

JITL-145 - Getting issue details... STATUS

JobScheduler Master Log Files in Detail

1. Debug Log: scheduler.log

The Debug Log is created:

• during JobScheduler startup. Just before startup the JobScheduler moves an existing log file scheduler.log to scheduler-old.log, and creates an empty scheduler.log file. At any time there will be only two Debug Log files, there is no archiving for Debug Log files.
• when log rotation is performed.
• by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent, not applicable for new JobScheduler Universal Agent .
• with the constant file name scheduler.log

The Debug Log:

• contains most of the information concerning internal working of the JobScheduler. 
• is subject to log rotation which archive existing file with suffix old e.g. scheduler-old.log and creates a new log file by the house keeping job sos/housekeeping/scheduler_rotate_log 
• is stored in the file system (the default location is ${SCHEDULER_DATA}/logs/).
• is always created. Users who wish not to use this log can apply the following settings to ./config/factory.ini that restrict the log to a minimum size:
  • Windows
    • log = nul
  • Unix
    • log = /dev/null

The Debug Log is presented in tabular form, with columns containing the following information:

• Column 1: Starts with "." if demo version, Day of Month
• Column 2: Time (with ms)
• Column 3: application name (Windows) - empty otherwise
• Column 4: elapsed time in ms
• Column 5: Process ID, ".", Thread ID (hex)
• Column 6: Used memory in MB (Windows) - empty otherwise
• Column 7: Log text

2. Main Log: scheduler-<yyyy-mm-dd-hhmmss>.<scheduler-id>.log

The Main Log is created:

• during JobScheduler startup.
• when log rotation is performed.
• by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent, not applicable for new JobScheduler Universal Agent .
• with the file name scheduler-<yyyy-mm-dd-hhmmss>.<scheduler-id>.log where
  • <yyyy-mm-dd-hhmmss> states the timestamp when the log was started.
  • <scheduler-id> states the JobScheduler instance identification that has been specified during installation. If multiple JobScheduler instances store log files to a common log folder then this is helpful to identify the log file of a specific JobScheduler instance.

The Main Log:

• has information about processes being executed by JobScheduler and other high level information of JobScheduler operation.
• is subject to log rotation which creates a new log file with a new timestamp by the house keeping job sos/housekeeping/scheduler_rotate_log  
• is stored on the file system (the default location is ${SCHEDULER_DATA}/logs/).

3. Job Log: job.<job_chain_name>,<job_name>.log

A Job Log is created:

• during start of JobScheduler for each job. Any job start will append an existing Job Log.

• when a job is added to a scheduling environment while the JobScheduler is running.
• with the file name job.<job_chain_name>,<job_name>.log where
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <job_name> is the name of the job, i.e. the <job_name> part in a <job_name>.job.xml job file.

The Job Log:

• is stored on the file system (the default location is ${SCHEDULER_DATA}/logs/).

4. Task Log: task.<job_chain_location>,<job_chain_name>,<job_name>.log

The Task Log is created:

• during execution of individual tasks (processes) for jobs. Any job start with a subsequent task will overwrite an existing Task Log.
• with the file name task.<job_chain_location>,<job_chain_name>,<job_name>.log where
  • <job_chain_location> is the folder name where the job chain is located.
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <job_name> is the name of the job, i.e. the <job_name> part in a <job_name>.job.xml job file.

The Task Log:

• contains all information about the execution of job instances, e.g. parameters, output to stdout and stderr of the script or application that is executed..
• is stored in the file system and - after execution - is saved to the database.

5. Order Log: order.<job_chain_location>,<job_chain_name>,<order_id>.log

The Order Log is created:

• during execution of an order for jobs in a job chain, i.e. when an order traverses from one node to another in a job chain.
• with the file name order.<job_chain_location>,<job_chain_name>,<order_id>.log where
  • <job_chain_location> is the folder name where the job chain is located.
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <order_id> is the identification of the order, i.e. the <order_id> part in a <job_chain_name>,<order_id>.order.xml order file.

The Order Log:

• contains all output of the individual Task Logs for jobs that the order passed when being executed in a job chain. This includes repeated execution of job nodes, e.g. to an order setback.
• is stored on the file system and - after execution - is saved to the database.

6. Installation Log: Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log

The Installation Log is created:

• when installing JobScheduler, this includes full installs and updates.
• with the file name Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log where 
  • <release_number> is the JobScheduler release number in the format <major>.<minor>.<bug-fix>
  • <yyyymmddhhmmss> states the timestamp when the installation was started.
  • <timestamp> states a more precise timestamp when the installation was performed.

The Installation Log:

• contains important information about JobScheduler installation. 
• is of vital importance in case that installation problems occur on a JobScheduler full installation or update.

7. Engine Log:  scheduler_engine.log

The JobScheduler by default does not create the Engine Log. This log has to be configured and is used primarily to debug JobScheduler engine actions.

• To create the scheduler_engine.log log file do following changes in the 
  • ${SCHEDULER_DATA}/config/factory.ini:  
    • options        = -DSCHEDULER_DATA="${SCHEDULER_DATA}" -Dlog4j.configuration="file:///${SCHEDULER_HOME}/lib/log4j.properties"
  • ${SCHEDULER_HOME}/lib/log4j.properties


    • log4j.logger.com.sos.scheduler.engine=debug, engine
log4j.appender.engine=org.apache.log4j.RollingFileAppender
log4j.appender.engine.File=${SCHEDULER_DATA}/logs/scheduler_engine.log
log4j.appender.engine.MaxFileSize=5000KB
log4j.appender.engine.MaxBackupIndex=30
log4j.appender.engine.layout=org.apache.log4j.PatternLayout
log4j.appender.engine.layout.ConversionPattern=%5p %-12d{HH:mm:ss,SSS} (%F:%L) - %m%n

      


• After updating both the files ${SCHEDULER_DATA}/config/factory.ini and ${SCHEDULER_HOME}/lib/log4j.properties restart the JobScheduler.

The Engine Log:

• contains important information about JobScheduler Engine actions.
• is stored on the file system.
• is not subject to log rotation by the rotate log job sos/housekeeping/scheduler_rotate_log.

8. Output Stream Log: scheduler.out

JS-1194 - Getting issue details... STATUS

The stdout/stderr output from JobScheduler is written on both Unix and Windows systems to a scheduler.out log file when the JobScheduler is running as a daemon or service.

The scheduler.out log file is created:

• during JobScheduler startup. Just before startup the JobScheduler creates an empty scheduler.log file. At any time there will be only one Output Stream Log file, old content is lost after the JobScheduler restarts.

The Output Stream Log:

• contains information that can be useful for debugging.
• is stored in the file system, the default location is ${SCHEDULER_DATA}/logs/..

9. Web Server Log: http.log

FEATURE AVAILABILITY ENDING WITH RELEASE 1.10.7

Note that this feature has been replaced by the JOC Cockpit, starting with JobScheduler Release 1.11.0. See the JOC Cockpit - Logging article for more information.

The Web Server Log is created when ever the JOC is accessed via Jetty Web Server e.g.  http://localhost:44440/jobscheduler/operations_gui/

The Web Server Log:

• contains access information from e.g. user's browser or from a web service client using POST/GET requests.
• its contains information such as browser version, OS version, user's IP address etc.
• is not subject to log rotation by the rotate log job sos/housekeeping/scheduler_rotate_log.

10. Web Service Log: web_service,<web_service_name>.<request_number>.log

The Web Service Log is created whenever JobScheduler is configured to accept SOAP requests and the setting debug=true is used for the web service in the scheduler.xml file.

For each web service and SOAP request a separate log file is created.

The log file name is made up of web_service,<web_service_name>.<request_number>.log where <web_service_name> is the respective name of the web service and <request_number> accordingly, example: web_service,scheduler.1.log.

The Web Service Log contains

• complete web service SOAP request.
• the JobScheduler XML command embedded in the SOAP request.
• the SOAP response sent back to the client with the status code and text.
• is not subject to log rotation by the rotate log job sos/housekeeping/scheduler_rotate_log.

This

JobScheduler Agent Log Files in Detail

Logging on the Master

JS-1497 - Getting issue details... STATUS

A Master JobScheduler logs tasks that are carried out on an Agent, together with the hostname of the Agent.

Logging on the Agent

For configuring log rotation and log compression see JobScheduler Universal Agent - Logging

1. Agent Main Log

By default you will find the log files for JobScheduler Universal Agent in the ${SCHEDULER_HOME}/logs folder where ${SCHEDULER_HOME} is the root directory of your JobScheduler Universal Agent installation.

The most recent Agent Main Log file is called jobscheduler_agent_<port>.log. where <port> ist the port that the Agent is listening to.

Log files are rotated on a daily basis should any job activity have occurred, so older log files are located in the ${SCHEDULER_HOME}/logs folder suffixed with a date.

The logging is configured in ${SCHEDULER_HOME}/lib/log4j.xml.

See also:

• JobScheduler Universal Agent - Installation & Operation

2. Agent Task Log

JS-1521 - Getting issue details... STATUS

JS-1523 - Getting issue details... STATUS

The Agent Task Log is created for each task that is executed with an Agent an consists of two files:

• task-<agentTaskID>-stdout.log: includes task output to stdout
• task-<agentTaskID>-stderr.log: includes task output to stderr

where <agentTaskID> is a unique identifier assigned to a task by the Agent.

The Agent Task Log files will be removed by the Agent after completion of a task. Should a task be killed by the Agent then the Agent Task Log will be retained for analysis purposes.

Logging Features

Log Rotation

JITL-95 - Getting issue details... STATUS

The JobScheduler creates the Main Log and Debug Log on start and restart. The JobScheduler also has a housekeeping job the sos/housekeeping/scheduler_rotate_log. It is considered good practice to configure scheduler_rotate_log job to run once every day, for details see JobSchedulerRotateLog. The scheduler_rotate_log job:

• creates a new Main Log  and Debug Log. 
• creates a archive copy of the existing Debug Log file before opening a new Debug Log.
  • saves an existing Debug Log to the file  scheduler-old.log. 

Its important to note that if the scheduler_rotate_log job is run every 24 hours, there will only be a maximum of 48 hours of Debug Log. 

Example:

If the scheduler_rotate_log job is being scheduled for a JobScheduler  with ID "SOS-BERLIN"  to run everyday early morning at 2am, then a new Main Log  will be created on 1st of June with time stamp eg. scheduler-2015-06-01-020000.SOS-BERLIN.log. On next day, 2nd of June, the Main Log file will will be rotated to the file name scheduler-2015-06-01-020000.SOS-BERLIN.log. 

if the JobScheduler is restarted on 3rd of June at 10:25:33 AM then a new Main Log will be created with the file name scheduler-2015-06-03-102533.SOS-BERLIN.log and again in the night a new log will be created by the scheduler_rotate_log job with the file name scheduler-2015-06-03-020000.SOS-BERLIN.log.

How to Find the Latest Log Files

For analysis of problems it is vital to know the exact point in time when an error occurs. You can find the point in time of an error e.g. from the JOC GUI:

• If the error occurred in a job chain then see the order history tab and open the corresponding Order Log. 
• If error has occurred in a stand alone job then check the task history tab and open the corresponding Task Log.

Then identify the according Main Log and Debug Log that covers the point in time of the error.

• The Debug Log uses a constant name scheduler.log.
  • If the housekeeping job scheduler_rotate_log has been run after the error occurred then the Debug Log has been rotated and the previous version is available with the file scheduler-old.log.
• The Main Log can be identified from the file name with the most recent time stamp before the point in time that the error occurred.

Finding Errors in the Debug Log and Main Log

Whenever there is an error or warning in the JobScheduler then the information will be logged to the Main Log and Debug Log.

All error and warning messages are written with the [ERROR] or [WARN] tags to the log file.

Start with the Main Log, open the file with any text editor and scan it from starting for any tail tail sign of the problem. You can search for the [ERROR] or [WARN] tags in the log. 

Log Level

Central Log Level settings 

The central Log Level setting affects the granularity of the information logged by JobScheduler in Debug Log and Main Log. The settings can be changed in the settings file $SCHEDULER_DATA/config/factory.ini. The default Log level is info. The Log Level can be one of error, warn, info, debug1 to debug9.

After changing the settings in the factory.ini file, the JobScheduler must be restarted to make settings effective.

The central Log Level settings will be applied to all the JobScheduler objects, e.g. jobs, job chains, orders etc.

log_level = debug9

Job Log Level settings

For debugging a specific job, a higher Log Level can be configured for the job. The Log Levels are the same as of the central Log Levels.

The configuration changes to a job do not require the JobScheduler to restart. After debugging the Log Level can be set back to info.

The JobScheduler Object Editor (JOE) provides a GUI to change the Log Level..Click on the job name in left panel of the JOE, switch to options tab in the right panel, change the Log Level settings to the desired Log Level. This would result in a job configuration like this:

<job order="yes"  stop_on_error="no"
     title="Simple execution of a shell script or a shell command with Log Level debug9">
    <settings >        
           <log_level ><![CDATA[debug9]]></log_level>
    </settings>
    <script language="shell">
        <![CDATA[
            echo . Hello World!!
        ]]>
    </script>
    <run_time/>
</job>

Confidentiality

Log output that is related to security credentials - user names, passwords and the location of private keys that are configured for JobScheduler operation - is filtered from log output.

This is done according to the type of credential as follows:

• passwords: hidden,
• user names: visible for developers/operators using debug levels,
• private key locations: - ditto -

JITL-145 - Getting issue details... STATUS

Viewing JobScheduler Log Files in the JOC Cockpit

In Addition to being read from the file System, a number of JobScheduler log files can be viewed by JOC Cockpit users:

• The debug and main logs for a JobScheduler Master (scheduler.log and scheduler-<yyyy-mm-dd-hhmmss>.<scheduler-id>.log respectively) can be downloaded from the JOC Cockpit's Dashboard view by JOC Cockpit users with permissions to view the JobScheduler Status widget in the Dashboard.
• Log files for individual Orders, Jobs and Tasks can be read in the relevant JOC Cockpit view by users with the necessary permissions. Permissions are described in detail in the Authentication and Authorization - Permissions for the JOC Cockpit Web Service article.

Note that log files can only be accessed via the JOC Cockpit when the JobScheduler Master in question does not have the status unreachable.

See the JOC Cockpit - Logging article for more Information.

References

Change Management References

See also

• How To - Logging
• JobScheduler Universal Agent - Logging

• How to configure JobScheduler for using log4j
• How to configure logging in JobScheduler
• How to limit the growth of the Main Log and Debug Log (scheduler.log)
• How to redirect error messages and warnings to the Windows Event Log
• How to send log output to a Syslog Server