What logging possibilities does JobScheduler provide?

Introduction

• The JobScheduler creates a number of different logs, to provide specific information about jobs, job chains, orders, tasks and JobScheduler internal operational itself.
• The location of the the log files can vary per installation. By default you will find the log files in the ${SCHEDULER_DATA}/logs folder where ${SCHEDULER_DATA} is the JobScheduler data directory that has been specified during installation. For Windows systems use %SCHEDULER_DATA% instead of ${SCHEDULER_DATA}.

JobScheduler log files

1. Debug Log: scheduler.log
2. Main Log: scheduler-yyyy-mm-dd-hhmmss.<schduler-id>.log
3. Job Log: job.job_chain_name,job_name.log
4. Task Log: task.job_chain_location,job_chain_name,job_name.log
5. Order Log: order.job_chain_location,job_chain_name,order_id.log
6. Installation Log: Install_VX.X.XXXX_timestamp.log
7. Web Server Log: http.log
8. Web Service Log: web_service,web_service_name.log

1. Debug Log: scheduler.log

The Debug Log is created

• during JobScheduler startup. Just before startup the JobScheduler moves an existing log file scheduler.log to scheduler-old.log, and creates an empty scheduler.log file. At any time there will be only two Debug Log files, there is no archiving for Debug Log files.
• when log rotation is performed.
• by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent.
• with the constant file name scheduler.log

The Debug Log 

• contains most of the information concerning internal working of the JobScheduler. 
• is subject to log rotation which creates a new log file with a new timestamp by the house keeping job sos/housekeeping/scheduler_rotate_log
• is stored in the file system.

Each line of the Debug Log states columns with the following information:

• Column 1: Starts with "." if demo version, Day of Month
• Column 2: Time (with ms)
• Column 3: application name (Windows) - empty otherwise
• Column 4: elapsed time in ms
• Column 5: Process ID, ".", Thread ID (hex)
• Column 6: Used memory in MB (Windows) - empty otherwise
• Column 7: Log text

2. Main Log: scheduler-<yyyy-mm-dd-hhmiss>.<scheduler-id>.log

The Main Log is created

• during JobScheduler startup.
• when log rotation is performed.
• by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent.
• with the file name scheduler-<yyyy-mm-dd-hhmmss>.<scheduler-id>.log where
  • <yyyy-mm-dd-hhmmss> states the timestamp when the log was started.
  • <scheduler-id> states the JobScheduler instance identification that has been specified during installation. If multiple JobScheduler instances store log files to a common log folder then this is helpful to identify the log file of a specific JobScheduler instance.

The Main Log 

• has information about processes being executed by JobScheduler and other high level information of JobScheduler operation.
• is subject to log rotation which creates a new log file with a new timestamp by the house keeping job sos/housekeeping/scheduler_rotate_log
• is stored in the file system.

3. Job Log: job.<job_chain_name>,<job_name>.log

The Job Log is created

• during execution of individual jobs. Any job start will overwrite an existing Job Log.
• with the file name job.<job_chain_name>,<job_name>.log where
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <job_name> is the name of the job, i.e. the <job_name> part in a <job_name>.job.xml job file.

The Job Log

• is stored in the file system and - after execution - is saved to the database.

4. Task Log: task.<job_chain_location>,<job_chain_name>,<job_name>.log

The Task Log is created

• during execution of individual tasks (processes) for jobs. Any job start with a subsequent task will overwrite an existing Task Log.
• with the file name task.<job_chain_location>,<job_chain_name>,<job_name>.log where
  • <job_chain_location> is the folder name where the job chain is located.
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <job_name> is the name of the job, i.e. the <job_name> part in a <job_name>.job.xml job file.

The Task Log

• contains all information about the execution of job instances, e.g. parameters, output to stdout and stderr of the script or application that is executed..
• is stored in the file system and - after execution - is saved to the database.

5. Order Log: order.<job_chain_location>,<job_chain_name>,<order_id>.log

The Order Log is created

• during execution of an order for jobs in a job chain, i.e. when an order traverses from one node to another in a job chain.
• with the file name order.<job_chain_location>,<job_chain_name>,<order_id>.log where
  • <job_chain_location> is the folder name where the job chain is located.
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <order_id> is the identification of the order, i.e. the <order_id> part in a <job_chain_name>,<order_id>.order.xml order file.

The Order Log

• contains all output of the individual Task Logs for jobs that the order passed when being executed in a job chain. This includes repeated execution of job nodes, e.g. to an order setback.
• is stored in the file system and - after execution - is saved to the database.

6. Installation Log: Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log

The Installation Log is created

• when installing JobScheduler, this includes full installs and updates.
• with the file name Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log where 
  • <release_number> is the JobScheduler release number in the format <major>.<minor>.<bug-fix>
  • <yyyymmddhhmmss> states the timestamp when the installation was started.
  • <timestamp> states a more precise timestamp when the installation was performed.

The Installation Log

• contains important information about JobScheduler installation. 
• is of vital importance in case that installation problems occur on a JobScheduler full installation or update.

Log Rotation 

The JobScheduler create Main Log and Debug Log when ever the JobScheduler is restated. The JobScheduler also has a housekeeping job the sos/housekeeping/scheduler_rotate_log. It is considered good practice to configure scheduler_rotate_log job to run every 24hours/once in the day. The scheduler_rotate_log job creates new Main Log  scheduler-<yyyy-mm-dd-hhmiss>.<scheduler-id>.log and Debug Log scheduler.log every time its runs. Also the scheduler_rotate_log job creates a archive copy of the existing Debug log file before opening a new Debug log.

When ever scheduler_rotate_log job is being executed existing debug log (scheduler.log) will be save to  scheduler-old.log. This its very important to note that if the scheduler_rotate_log job is running every 24 hours, at any point there is only 48 hours of debug log. 

If the scheduler_rotate_log job is being scheduled for JobScheduler JS-SOS-BERLIN-GOLLUM  to run everyday early morning at 02:00:00 am, a new Main Log  will be created on 1st of June with time stamp eg. scheduler-2015-06-01-020000.JS-SOS-BERLIN-GOLLUM.log, next day on next day 2nd of June Main Log file will have name scheduler-2015-06-01-020000.JS-SOS-BERLIN-GOLLUM.log. 

if the JobScheduler is restarted on 3rd of June at 10:25:33 AM new Main Log will be created with name scheduler-2015-06-03-102533.JS-SOS-BERLIN-GOLLUM.log and again in the night a new log will be created by the scheduloer_rotate_log job with the name scheduler-2015-06-03-020000.JS-SOS-BERLIN-GOLLUM.log.

How to Find the Latest Logs

Once you have the time of error, next step is to find the correct Main Log and Debug Log. If error has occurred after the scheduler_rotate_log has be executed most likely information corresponding to error will be in the latest Main Log and Debug Log scheduler.log.

if error has occurred before the housekeeping job scheduler_rotate_log has been run, check the Main Log from day before and check the archived Debug Log scheduler-old.log. 

Finding Error in the Debug Log and Main Log

When ever there is a error or warning in the JobScheduler lots of information is being logged into Main Log and Debug Log. Before starting analysis of the Logs note down the point in time of the error. One can find point in time of the error from the JOC GUI. If error occurred in a job chain see the order history tab and open the corresponding order log, but if error has occurred in the stand alone job check the task log and open the corresponding task log. 

All the error and warning message are written with the [ERROR] and [WARN] tags in the task log, Main Log and Debug Log. 

Start with the Debug Log scheduler.log file, open the file with any text editor (Ultraedit /Vim) and scan it from starting for any tail tail sign of the problem. see if there is any message regarding dropped database connection, the JobScheduler's port is blocked, not enough disk space etc. One can search for [Error] or [WARN] text in the logs. 

To get the complete information about the cause and nature of the error its important to correlate Debug Log and Main Log and see what was the root cause of the problem. Most of the time root cause of the error will appear in the log before or at the time of the actual error.

Hints

• The Job Log, Task Log and Order Log files are stored in the file system and saved in the database and can be accessed from the JOC and JID components.
• All log files are stored in the file system in the ${SCHEDULER_DATA}/logs folder. For Windows systems use %SCHEDULER_DATA% instead of ${SCHEDULER_DATA}.