JOC Cockpit - Audit Log

Introduction

The Audit Log allows compliance and operations reporting requirements for job scheduling activities to be met by providing an automatic and secure record of all actions that users of the JobScheduler Web Service - including JOC Cockpit users and external applications - perform on job-related objects. The Audit Log lists the relevant JobScheduler objects - Jobs, Job Chains, Orders, etc. - and any actions that modify their status.The Audit Log is written by the JobScheduler Web Service as shown in the JOC Cockpit - Architecture page.

Scope of the Audit Log

The Audit Log is maintained for all job-related objects such as Jobs, Job Chains, Orders, etc., whose statuses are modified by users carrying out actions such as starting, stopping or pausing an Order or Job. It uses a format that is human and machine readable and fulfills the following main functions:

• provide a compliance-conform record in a simple file format that can be archived and handed over to a compliance officer,
• provide operational records of job scheduling activities for reporting purposes,
• provide an operational record that is readily accessible to operating staff,
• provide a second source of operational information if the database used to store operational records is not available.

Users of the JOC Cockpit with the necessary permissions can view the Audit Log in the JOC Cockpit by clicking the Audit Log link in the JOC Cockpit main menu as shown in the screenshot below. This view and, in particular, the Reasons, Time Spent and Ticket Link information is intended to provide users with more relevant operational information than is available from the History.

Caveat

The Audit Log contains information about changes to JobScheduler objects initiated from the JOC Cockpit and by other applications using the JobScheduler Web Services. It does not include changes made to objects using the JobScheduler Object Editor, JOE, or changes made directly to objects on the file system using, for example, a text editor. 

Description

Compliance, Reporting and Operational Features

• The Audit Log is written automatically by the JobScheduler Web Services:
  • Entries to the Audit log are made when an action that changes the state of a JobScheduler Object is initiated in the JOC Cockpit or other application accessing the JobScheduler Web Services.
  •  There is no mechanism for switching off the Audit Log.
• The Audit Log is written in two locations simultaneously:
  • In the DBMS used by the JobScheduler Web Services: 
    • This instance is used by the JOC Cockpit to allow operators to view the Audit Log information.
    • It is also used by the JOC Cockpit to generate a Microsoft Excel^®-compatible export file for reporting purposes.
  • In the Audit Log file:
    • It is intended that this instance is archived and used as a compliance record. System Administrators can make a copy of this instance available to operating staff if the DBMS should not be available. 
    • The Audit Log file is named JOCAuditLog.log and saved in
      • the jetty_base/logs folder on Linux systems and on Windows where jetty_base is the directory containing the Jetty web server configuration files as described in the JOC Cockpit - Installation article.
• All new entries will be added to the Audit Log - neither the log file not the DMBS records are overwritten.
  • System administrators have to arrange an archiving and housekeeping system for the Audit Log file according to relevant compliance laws and corporate governance.
  • It is the responsibility of system administrators to check compliance requirements such as the retention period for the information contained in the Audit Log file.
• The Audit Log is visible in the JOC Cockpit to:
  • Users with the appropriate permissions. The specification of user permissions is described in the Authentication and Authorization - Configuration article. 
    
    • The default authorization permissions for the Audit Log which are configured for the JOC Cockpit are listed in the Permissions Matrix shown in the Authentication and Authorization - Permissions for the JOC Cockpit Web Service article.  
  • System Administrators with access to the DBMS used by the JOC Cockpit or the file system where the Audit Log file is written.
• The Audit Log is read-only for all JOC Cockpit users - only System Administrators that have access to both the file system and to the database can modify the log. 

Key to the Audit Log Entries

• Created: the point in time when the action was performed.
• Account: the user account that caused the action.
• Request URL: the Web Service URL that was invoked by the JOC Cockpit. The URLs represent the object type and action, e.g. /orders/start.
• Job Chain, Order ID, Job: identifies the object for which the action was performed.
• Reasons: (can be mandatory or optional - see below) an explanation of why an action was carried out. Reasons are created either by selecting an item from a list of predefined reasons (configurable) or individual - i.e. free text input. A maximum of 2000 characters can be added to a free text reason.
• Time Spent: (optional) the number of minutes required for the action (including time for impact analysis) can be added.
• Ticket Link: (optional) a URL can be added that points to a ticket system, e.g. the SOS Ticket System, to identify the underlying ticket for the intervention.

Action Comments: Reasons, Time Spent and Ticket Link

Users carrying out actions are able to add an explanatory comments - Reason, Time Spent and Ticket Link - for each action. These Comments are entered as part of the specification of an action and then saved in the Audit Log. They can then be read by all other JOC Cockpit users that have the necessary permissions. The form for the 'Start Order At' action is shown as an example in the following screenshot:

Action Comments are made up of three elements as shown in the screenshot above:

• Reasons (mandatory or optional):
  • Predefined Reasons which are selected from a drop-down list or 
  • Individual Reasons, which are entered in a text field
• Time Spent (always optional):
  • in minutes
• Ticket Link (always optional):
  • has to start with http://

Reasons are intended as an aid to later understanding why an action was carried out - for example, for operators on later shifts or for reporting purposes.

The Time Spent and Ticket Link parameters are intended as basic information for accounting and reporting which, if required, can be made transparent for job scheduling end customers.  

Mandatory or Optional Reasons

Reasons can be mandatory or optional for all users of the JOC Cockpit.

• If Reasons are mandatory:
  • Users must provide a Reason for each action they carry out.
  • Users will be offered the option of entering Time Spent and Ticket Link parameters.
• If Reasons are optional:
  • Users still have the possibility of activating them on an individual basis in their User Profile.
    • Activation of Reasons also allows users to enter Time Spent and Ticket Link parameters.
  • Reasons, along with Time Spent and Ticket Link parameters will be added to the Audit Log.

Configuring Mandatory Reasons

The  force_comments_for_audit_log flag is used to make the entry of Reasons mandatory. This parameter is set in the joc.properties file as shown in the following code box:

### This flag controls if the comment is required or not.
force_comments_for_audit_log = false

The joc.properties file is located in the ./joc_home/jetty_base/resources/joc directory. 

The default setting for force_comments_for_audit_log is false .

Either predefined or individual Reasons can be set when force_comments_for_audit_log = true is set.

Optional Action Notes

If Reasons are not mandatory - i.e. the force_comments_for_audit_log flag is set to false -  users can configure whether they wish to be offered the option of entering Reasons for the actions they carry out. This is configured with the Enable Reasons for Audit Log setting in Users' profiles as shown in the next screenshot:

If this option is not set and Reasons are not mandatory, which are the default settings, then Users will not be presented with the Enter a Reason function when starting Orders or Jobs.

Behavior of Start Order now and Start Task now 

The behavior of the Start Order now and Start Task now functions depends on whether or not Reasons are enabled in the user's Profile setting and whether Reasons are mandatory or optional (i.e. whether force_comments_for_audit_log = true or false).

Predefined Reasons

A predefined set of reasons is delivered with the JOC Cockpit.

Predefined Reasons are found in the ./joc_home/jetty_base/resources/joc directory, in the joc.properties file. 

The predefined Reasons are listed in the following code box:

### Pre-defined comments
comments =  System maintenance; \
            Repeat execution; \
            Business requirement; \
            Restart failed execution; \
            Re-instantiate stopped object; \
            Temporary stop; \
            Change of JobScheduler object; \
            Rerun with parameter changes; \
            Change of external dependency; \
            Application deployment and upgrade

Predefined Reasons Syntax

• The ";" character near the end of each line marks the end of the comments. The semicolon here functions as a delimiter and allows commas to be used in reasons.
• The "\" character marks the end of a line.

Modifying Reasons Settings

System administrators are free to modify predefined reasons and the force_comments_for_audit_log flag and to modify the predefined Reasons to suit their own requirements.

The JOC Cockpit needs to be restarted before changes made to the joc.properties file will take effect.

Individual Reasons

Individual Reasons can be entered in the text field as shown in the screenshot near the top of this page. 

UTF-8 characters are allowed.

Audit Log View

The complete Audit Log can be viewed by selecting Audit Log in the JOC Cockpit Main Menu.

An extract from the Audit Log for an individual Job Chain can be seen in the Job Chains view when Show Job Chain Details is selected. This is not available for standalone Jobs. 

The Request Body

The arrow at the left hand side of each entry in the Audit Log view list allows more detailed information about the request submitted to the JobScheduler Web Services to be seen - the Request Body. This opens in a new row in the Audit Log as shown in the following screenshot and includes information about the Job Scheduler which the Order is to be processed on, as well as the Order Parameters and starting time (processing starting from State 20, starting immediately, etc):

Filters

Filters are available to allow selected entries from the Audit Log to be displayed.

The Advanced Filters

In addition to the preset All, Today, Last hour, etc. filters listed at the top of the Audit Log view, there is an Advanced Filter function which is opened with the Advanced Filter button at the top right of the view. Clicking this button opens the Advanced Filter function. Note that the last filter selected overwrites any previous filters. For example, if the Today preset filter is selected and then From 09:00:00 To 18:00:00 then all Audit Log entries started between 09:00 and 18:00 will be shown for all the days covered by the Audit Log.

The Search Field

The Search field allows a search of the results of the preset or Advanced Filters to be carried out. 

Audit Log Location

The Audit Log file is found in the ./joc_home/jetty_base/logs directory, in the JOCAuditLog.log file. If the file does not exist, it will be created with the first action modifying the status of an object that is carried out by a user.

The location and name of the Audit Log file are specified in a log4j.properties file. See the Logging Configuration section of the JOC Cockpit - Logging article for more information.

Audit Log Sample

The following sample shows a number of Web Service requests that have been performed by two users, aa_ito and root, who logged in at 14:13:47 in parallel to aa_ito using a different browser :

2017-02-10 11:39:55,992 INFO  REQUEST: ./orders/add - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"jobChain":"/tutorials/hello_world_job_chain","at":"now"}]} - COMMENT: Demo start for documentation
2017-02-10 11:43:05,831 INFO  REQUEST: ./jobs/start - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","jobs":[{"job":"/standalone-jobs/test_sa_job1","at":"now"}]} - COMMENT: Demo start for documentation
2017-02-10 11:54:11,731 INFO  REQUEST: ./jobs/start - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","jobs":[{"job":"/standalone-jobs/test_sa_job1","at":"now"}]} - COMMENT: null
2017-02-10 12:38:31,507 INFO  REQUEST: ./login - USER: aa_ito - PARAMS: {} - COMMENT: it_operator
2017-02-10 12:48:39,857 INFO  REQUEST: ./orders/start - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"orderId":"hello_world_order","jobChain":"/tutorials/hello_world_job_chain","at":"now"}]} - COMMENT: null
2017-02-10 13:34:57,995 INFO  REQUEST: ./login - USER: aa_ito - PARAMS: {} - COMMENT: 
2017-02-10 13:35:03,391 INFO  REQUEST: ./login - USER: aa_ito - PARAMS: {} - COMMENT: it_operator
2017-02-10 13:35:49,186 INFO  REQUEST: ./orders/start - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"orderId":"hello_world_order","jobChain":"/tutorials/hello_world_job_chain","at":"now"}]} - COMMENT: null
2017-02-10 13:36:13,905 INFO  REQUEST: ./jobs/start - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","jobs":[{"job":"/standalone-jobs/test_sa_job1","at":"now"}]} - COMMENT: null
2017-02-10 13:53:50,184 INFO  REQUEST: ./orders/add - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"jobChain":"/tutorials/hello_world_job_chain","state":"20","at":"now"}]} - COMMENT: Test Start from step 20
2017-02-10 13:56:05,459 INFO  REQUEST: ./orders/add - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"jobChain":"/tutorials/hello_world_job_chain","at":"now"}]} - COMMENT: null
2017-02-10 14:02:30,467 INFO  REQUEST: ./login - USER: aa_ito - PARAMS: {} - COMMENT: it_operator
2017-02-10 14:09:38,471 INFO  REQUEST: ./orders/add - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"jobChain":"/tutorials/hello_world_job_chain","state":"20","at":"now"}]} - COMMENT: Repeat execution
2017-02-10 14:10:36,839 INFO  REQUEST: ./orders/add - USER: aa_ito - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"jobChain":"/tutorials/hello_world_job_chain","state":"20","at":"now"}]} - COMMENT: Repeat execution
2017-02-10 14:13:47,553 INFO  REQUEST: ./login - USER: root - PARAMS: {} - COMMENT: all
2017-02-10 14:14:48,351 INFO  REQUEST: ./orders/add - USER: root - PARAMS: {"jobschedulerId":"jobscheduler_1.11","orders":[{"jobChain":"/tutorials/hello_world_job_chain","state":"20","at":"now"}]} - COMMENT: Repeat execution

Note that the COMMENT is used in the login REQUEST to show the Role a user has.

Audit Log Explanation

• The Audit Log includes the Web Service URL, which allows the object type and operation to be identified. For example, ./orders/resume translates to the action of resuming an order that is identified by its order ID and Job Chain path, as specified in the PARAMS part of each log entry.

Audit Log Export

The Audit Log Export function, which is activated with the button with the arrow shown in the next screenshot, allows the Audit Log to be exported in a reporting-friendly format either in whole or filtered.

The filtered table as shown in Audit Log view is exported as a Microsoft Excel^® compatible .xls file. The next screenshot shows an exported file (opened in Open Office Calc).

Note that the PARAMS part of records (see the Audit Log Sample section above) are not included in the export.

Export Configuration

The export function only takes the Audit Log entries that are shown in the Audit Log view. In order to export an Audit Log with, for example 900 entries, all 900 entries have to be loaded into the Audit Log view and the pagination set to 1000 as shown in the next screenshot:

In order to export an Audit Log of, for example, 2500 entries, the Max. number of entries per page setting in the User Profile should first be increased - in this case 3000 would be sufficient.

It will now be possible to set the pagination to this increased value as shown in the next screenshot:

Note that the Max. number of entries per page setting should be reset after the export has been carried out as it may otherwise considerably slow down loading of other list views such as the History.

Audit Log Housekeeping & Archiving

As already noted the JOC Cockpit adds to the Audit Log on an ongoing basis.

System administrators need to implement their own housekeeping / log rotation and archiving system according to their business and compliance requirements.