What logging possibilities does JobScheduler provide?

Introduction

• The JobScheduler creates a number of different logs, to provide specific information about jobs, job chains, orders, tasks and JobScheduler internal operational itself.
• The location of the the log files can vary per installation. By default you will find the log files in the ${SCHEDULER_DATA}/logs folder where ${SCHEDULER_DATA} is the JobScheduler data directory that has been specified during installation. For Windows systems use %SCHEDULER_DATA% instead of ${SCHEDULER_DATA}.

JobScheduler log files

1. Debug Log: scheduler.log
2. Main Log: scheduler-yyyy-mm-dd-hhmmss.<schduler-id>.log
3. Job Log: job.job_chain_name,job_name.log
4. Task Log: task.job_chain_location,job_chain_name,job_name.log
5. Order Log: order.job_chain_location,job_chain_name,order_id.log
6. Installation Log: Install_VX.X.XXXX_timestamp.log
7. Web Server Log: http.log
8. Web Service Log: web_service,web_service_name.log

1. Debug Log: scheduler.log

The Debug Log is created

• during JobScheduler startup. Just before startup the JobScheduler moves an existing log file scheduler.log to scheduler-old.log, and creates an empty scheduler.log file. At any time there will be only two Debug Log files, there is no archiving for Debug Log files.
• when log rotation is performed.
• by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent.
• with the constant file name scheduler.log

The Debug Log 

• contains most of the information concerning internal working of the JobScheduler. 
• is subject to log rotation which creates a new log file with a new timestamp by the house keeping job sos/housekeeping/scheduler_rotate_log
• is stored in the file system.

Each line of the Debug Log states columns with the following information:

• Column 1: Starts with "." if demo version, Day of Month
• Column 2: Time (with ms)
• Column 3: application name (Windows) - empty otherwise
• Column 4: elapsed time in ms
• Column 5: Process ID, ".", Thread ID (hex)
• Column 6: Used memory in MB (Windows) - empty otherwise
• Column 7: Log text

2. Main Log: scheduler-<yyyy-mm-dd-hhmiss>.<scheduler-id>.log

The Main Log is created

• during JobScheduler startup.
• when log rotation is performed.
• by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent.
• with the file name scheduler-<yyyy-mm-dd-hhmmss>.<scheduler-id>.log where
  • <yyyy-mm-dd-hhmmss> states the timestamp when the log was started.
  • <scheduler-id> states the JobScheduler instance identification that has been specified during installation. If multiple JobScheduler instances store log files to a common log folder then this is helpful to identify the log file of a specific JobScheduler instance.

The Main Log 

• has information about processes being executed by JobScheduler and other high level information of JobScheduler operation.
• is subject to log rotation which creates a new log file with a new timestamp by the house keeping job sos/housekeeping/scheduler_rotate_log
• is stored in the file system.

3. Job Log: job.<job_chain_name>,<job_name>.log

The Job Log is created

• during execution of individual jobs. Any job start will overwrite an existing Job Log.
• with the file name job.<job_chain_name>,<job_name>.log where
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <job_name> is the name of the job, i.e. the <job_name> part in a <job_name>.job.xml job file.

The Job Log

• is stored in the file system and - after execution - is saved to the database.

4. Task Log: task.<job_chain_location>,<job_chain_name>,<job_name>.log

The Task Log is created

• during execution of individual tasks (processes) for jobs. Any job start with a subsequent task will overwrite an existing Task Log.
• with the file name task.<job_chain_location>,<job_chain_name>,<job_name>.log where
  • <job_chain_location> is the folder name where the job chain is located.
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <job_name> is the name of the job, i.e. the <job_name> part in a <job_name>.job.xml job file.

The Task Log

• contains all information about the execution of job instances, e.g. parameters, output to stdout and stderr of the script or application that is executed..
• is stored in the file system and - after execution - is saved to the database.

5. Order Log: order.<job_chain_location>,<job_chain_name>,<order_id>.log

The Order Log is created

• during execution of an order for jobs in a job chain, i.e. when an order traverses from one node to another in a job chain.
• with the file name order.<job_chain_location>,<job_chain_name>,<order_id>.log where
  • <job_chain_location> is the folder name where the job chain is located.
  • <job_chain_name> is the name of the job chain that the job is executed for, i.e. the <job_chain_name> part of a <job_chain_name>.job_chain.xml job chain file.
  • <order_id> is the identification of the order, i.e. the <order_id> part in a <job_chain_name>,<order_id>.order.xml order file.

The Order Log

• contains all output of the individual Task Logs for jobs that the order passed when being executed in a job chain. This includes repeated execution of job nodes, e.g. to an order setback.
• is stored in the file system and - after execution - is saved to the database.

6. Installation Log: Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log

The Installation Log is created

• when installing JobScheduler, this includes full installs and updates.
• with the file name Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log where 
  • <release_number> is the JobScheduler release number in the format <major>.<minor>.<bug-fix>
  • <yyyymmddhhmmss> states the timestamp when the installation was started.
  • <timestamp> states a more precise timestamp when the installation was performed.

The Installation Log

• contains important information about JobScheduler installation. 
• is of vital importance in case that installation problems occur on a JobScheduler full installation or update.

Log Rotation 

The JobScheduler creates the Main Log and Debug Log on start and restart. The JobScheduler also has a housekeeping job the sos/housekeeping/scheduler_rotate_log. It is considered good practice to configure scheduler_rotate_log job to run once in the day. The scheduler_rotate_log job 

• creates a new Main Log  and Debug Log. 
• creates a archive copy of the existing Debug Log file before opening a new Debug Log.
• saves an existing Debug Log to the file  scheduler-old.log. This its very important to note that if the scheduler_rotate_log job is running every 24 hours, at any point there is only 48 hours of Debug Log. 

Example:

If the scheduler_rotate_log job is being scheduled for a JobScheduler  with ID "SOS-BERLIN"  to run everyday early morning at 2am, then a new Main Log  will be created on 1st of June with time stamp eg. scheduler-2015-06-01-020000.SOS-BERLIN.log. On next day, 2nd of June, the Main Log file will will be rotated to the file name scheduler-2015-06-01-020000.SOS-BERLIN.log. 

if the JobScheduler is restarted on 3rd of June at 10:25:33 AM then a new Main Log will be created with the file name scheduler-2015-06-03-102533.SOS-BERLIN.log and again in the night a new log will be created by the scheduler_rotate_log job with the file name scheduler-2015-06-03-020000.SOS-BERLIN.log.

How to Find the Latest Log Files

For analysis of problems it is vital to know the exact point in time when an error occurs. You can find the point in time of an error e.g. from the JOC GUI:

• If the error occurred in a job chain then see the order history tab and open the corresponding Order Log. 
• If error has occurred in a stand alone job then check the task history tab and open the corresponding Task Log.

Then identify the according Main Log and Debug Log that covers the point in time of the error.

• The Debug Log uses a constant name scheduler.log.
  • If the housekeeping job scheduler_rotate_log has been run after the error occurred then the Debug Log has been rotated and the previous version is available with the file scheduler-old.log.
• The Main Log can be identified from the file name with the most recent time stamp before the point in time that the error occurred.

Finding Errors in the Debug Log and Main Log

Whenever there is an error or warning in the JobScheduler then the information will be logged to the Main Log and Debug Log.

All error and warning messages are written with the [ERROR] or [WARN] tags to the log file.

Start with the Main Log, open the file with any text editor and scan it from starting for any tail tail sign of the problem. You can search for the [ERROR] or [WARN] tags in the log.