Introduction
- The JobScheduler creates a number of different logs, to provide specific information about jobs, job chains, orders, tasks and JobScheduler internal operational itself.
- The location of the the log files can vary per installation. By default you will find the log files in the
${SCHEDULER_DATA}/logs
folder where${SCHEDULER_DATA}
is the JobScheduler data directory that has been specified during installation. For Windows systems use%SCHEDULER_DATA%
instead of${SCHEDULER_DATA}
.
JobScheduler log files
- Debug Log: scheduler.log
- Main Log: scheduler-yyyy-mm-dd-hhmmss.<schduler-id>.log
- Job Log: job.job_chain_name,job_name.log
- Task Log: task.job_chain_location,job_chain_name,job_name.log
- Order Log: order.job_chain_location,job_chain_name,order_id.log
- Installation Log: Install_VX.X.XXXX_timestamp.log
- Web Server Log: http.log
- Web Service Log: web_service,web_service_name.log
1. Debug Log: scheduler.log
The Debug Log is created
- during JobScheduler startup. Just before startup the JobScheduler moves an existing log file
scheduler.log
toscheduler-old.log
, and creates an emptyscheduler.log
file. At any time there will be only two Debug Log files, there is no archiving for Debug Log files. - when log rotation is performed.
- by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent.
- with the constant file name
scheduler.log
The Debug Log
- contains most of the information concerning internal working of the JobScheduler.
- is subject to log rotation which creates a new log file with a new timestamp by the house keeping job
sos/housekeeping/scheduler_rotate_log
- is stored in the file system.
Each line of the Debug Log states columns with the following information:
- Column 1: Starts with "." if demo version, Day of Month
- Column 2: Time (with ms)
- Column 3: application name (Windows) - empty otherwise
- Column 4: elapsed time in ms
- Column 5: Process ID, ".", Thread ID (hex)
- Column 6: Used memory in MB (Windows) - empty otherwise
- Column 7: Log text
2. Main Log: scheduler-<yyyy-mm-dd-hhmiss>.<scheduler-id>.log
The Main Log is created
- during JobScheduler startup.
- when log rotation is performed.
- by all types of JobScheduler instances, i.e. Standalone, Active Cluster, Passive Cluster, Agent.
- with the file name
scheduler-<yyyy-mm-dd-hhmmss>.<scheduler-id>.log
where<yyyy-mm-dd-hhmmss>
states the timestamp when the log was started.<scheduler-id>
states the JobScheduler instance identification that has been specified during installation. If multiple JobScheduler instances store log files to a common log folder then this is helpful to identify the log file of a specific JobScheduler instance.
The Main Log
- has information about processes being executed by JobScheduler and other high level information of JobScheduler operation.
- is subject to log rotation which creates a new log file with a new timestamp by the house keeping job
sos/housekeeping/scheduler_rotate_log
- is stored in the file system.
3. Job Log: job.<job_chain_name>,<job_name>.log
The Job Log is created
- during execution of individual jobs. Any job start will overwrite an existing Job Log.
- with the file name
job.<job_chain_name>,<job_name>.log
where<job_chain_name>
is the name of the job chain that the job is executed for, i.e. the<job_chain_name>
part of a<job_chain_name>.
job_chain.xm
l
job chain file.<job_name>
is the name of the job, i.e. the<job_name>
part in a<job_name>.job.xml
job file.
The Job Log
- is stored in the file system and - after execution - is saved to the database.
4. Task Log: task.<job_chain_location>,<job_chain_name>,<job_name>.log
The Task Log is created
- during execution of individual tasks (processes) for jobs. Any job start with a subsequent task will overwrite an existing Task Log.
- with the file name
task.<job_chain_location>,<job_chain_name>,<job_name>.log
where<job_chain_location>
is the folder name where the job chain is located.<job_chain_name>
is the name of the job chain that the job is executed for, i.e. the<job_chain_name>
part of a<job_chain_name>.
job_chain.xm
l
job chain file.<job_name>
is the name of the job, i.e. the<job_name>
part in a<job_name>.job.xml
job file.
The Task Log
- contains all information about the execution of job instances, e.g. parameters, output to stdout and stderr of the script or application that is executed..
- is stored in the file system and - after execution - is saved to the database.
5. Order Log: order.<job_chain_location>,<job_chain_name>,<order_id>.log
The Order Log is created
- during execution of an order for jobs in a job chain, i.e. when an order traverses from one node to another in a job chain.
- with the file name
order.<job_chain_location>,<job_chain_name>,<order_id>.log
where<job_chain_location>
is the folder name where the job chain is located.<job_chain_name>
is the name of the job chain that the job is executed for, i.e. the<job_chain_name>
part of a<job_chain_name>.
job_chain.xm
l
job chain file.<order_id>
is the identification of the order, i.e. the<order_id>
part in a<job_chain_name>,<order_id>.order.xml
order file.
The Order Log
- contains all output of the individual Task Logs for jobs that the order passed when being executed in a job chain. This includes repeated execution of job nodes, e.g. to an order setback.
- is stored in the file system and - after execution - is saved to the database.
6. Installation Log: Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log
The Installation Log is created
- when installing JobScheduler, this includes full installs and updates.
- with the file name Install_V<release_number>_<yyyymmddhhmmss>_<timestamp>.log where
<release_number>
is the JobScheduler release number in the format<major>.<minor>.<bug-fix>
<yyyymmddhhmmss>
states the timestamp when the installation was started.<timestamp>
states a more precise timestamp when the installation was performed.
The Installation Log
- contains important information about JobScheduler installation.
- is of vital importance in case that installation problems occur on a JobScheduler full installation or update.
Log Rotation
The JobScheduler create Main Log and Debug Log when ever the JobScheduler is restated. The JobScheduler also has a housekeeping job the sos/housekeeping/scheduler_rotate_log.
It is considered good practice to configure scheduler_rotate_log job to run every 24hours/once in the day. The scheduler_rotate_log
job creates new Main Log scheduler-<yyyy-mm-dd-hhmiss>.<scheduler-id>.log
and Debug Log scheduler.log
every time its runs. Also the scheduler_rotate_log job creates a archive copy of the existing Debug log file before opening a new Debug log.
When ever scheduler_rotate_log
job is being executed existing debug log (scheduler.log
) will be save to scheduler-old.log
. This its very important to note that if the scheduler_rotate_log
job is running every 24 hours, at any point there is only 48 hours of debug log.
If the scheduler_rotate_log job is being scheduled for JobScheduler JS-SOS-BERLIN-GOLLUM to run everyday early morning at 02:00:00 am, a new Main Log will be created on 1st of June with time stamp eg. scheduler-2015-06-01-020000.JS-SOS-BERLIN-GOLLUM.log
, next day on next day 2nd of June Main Log file will have name scheduler-2015-06-01-020000.JS-SOS-BERLIN-GOLLUM.log.
if the JobScheduler is restarted on 3rd of June at 10:25:33 AM new Main Log will be created with name scheduler-2015-06-03-102533.JS-SOS-BERLIN-GOLLUM.log and again in the night a new log will be created by the scheduloer_rotate_log job with the name scheduler-2015-06-03-020000.JS-SOS-BERLIN-GOLLUM.log.
Finding Error in the Debug Log and Main Log
When ever there is a error or warning in the JobScheduler lots of information is being logged into Main Log and Debug Log. Before starting analysis of the Logs note down the point in time of the error. One can find point in time of the error from the JOC GUI. If error occurred in a job chain see the order history tab and open the corresponding order log, but if error has occurred in the stand alone job check the task log and open the corresponding task log.
All the error and warning message are written with the [ERROR] and [WARN] tags in the task log, Main Log and Debug Log.
Once you have the time of error, next step is to find the correct Main Log and Debug Log. If error has occurred after the scheduler_rotate_log has be executed most likely information corresponding to error will be in the lates Main Log and Debug Log scheduler.log.
Start with the Debug Log scheduler.log file, open the file with any text editor (Ultraedit /Vim) and scan it from starting for any tail tail sign of the problem. see if there is any message regarding dropped database connection, the JobScheduler's port is blocked, not enough diskspace etc. One can search for [Error] or [WARN] text in the logs.
To get the complete information about the cause and nature of the error its important to correlate Debug Log and Main Log and see what
Hints
- The Job Log, Task Log and Order Log files are stored in the file system and saved in the database and can be accessed from the JOC and JID components.
- All log files are stored in the file system in the
${SCHEDULER_DATA}/logs
folder. For Windows systems use%SCHEDULER_DATA%
instead of${SCHEDULER_DATA}
.