...

• Streaming makes job log output available to JOC Cockpit within 1s. 
• Streaming allows a running log which translates to the fact that users observe ongoing log output of orders and jobs in the log view window.
• In case of missing connections:
  • between Controller and Agent - the Agent stores log output in its journal and forwards logs when reconnected to a Controller later on. 
  • between JOC Cockpit and Controller - the Controller stores log output in its journal and forwards logs when reconnected to JOC Cockpit at a later point in time.

Log Size

Jobs can write log output of arbitrary size to the job's stdout/stderr channel.

• The JS7 Agent and Controller handle log output based on a streaming implementation that can handle any size of log output.
• The JOC Cockpit adds log output to the JS7 database in a compressed format. This processing is affected by large log size by the following aspects:
  • The compression ratio depends on the individual log output, however, it is safe to assume the resulting size of logs in the database to be 10-20% of the original size.
  • The compression process takes more time and more system resources such as CPU and Java Heap Space for larger log output.

The JS7 - History Service that handles logs in JOC Cockpit works with the following threshold settings, see JS7 - Settings:

• A setting for the applicable size of a task log (name: log_applicable_size, default: 500 MB). A task log that exceeds this size will be handled like this:
  • The first 100 KB and the last 100 KB of log output will be used as the task log.
    • A banner information is added to the begin of log output:
      • LOG OUTPUT EXCEEDS APPLICABLE SIZE OF <log_applicable_size> MB AND IS TRUNCATED TO THE FIRST AND LAST 100 KB.
    • The truncated task log will be compressed and added to the database.
  • The original log file will be removed.
  • Occurrence of log output exceeding this threshold raises a warning, however, the task execution history and truncated log output are available from the JOC Cockpit API and GUI.
• A setting for the maximum size of a task log (name: log_maximum_size, default: 1000 MB). A task log exceeding this size will be handled like this:
  • The History Service will stop to write larger log files. Log output exceeding the threshold value is ignored.
  • The first 100 KB of the log file will be used as the task log.
    • A banner information is added to the begin of log output:
      • LOG OUTPUT EXCEEDS MAXIMUM SIZE OF <log_maximum_size> MB AND IS TRUNCATED TO 100 KB.
    • The truncated task log will be compressed and added to the database.
  • The original log file will be removed.

The History Service similarly considers the above settings when performing switch-over and fail-over operations between JOC Cockpit instances (compressing and storing current logs to the database).

The threshold settings can be modified from the JOC Cockpit Settings page.

Log Files

A number of users prefer to have log files available for each order execution and job execution, for example to consolidate log files on a file server or to submit log files to specific tools for log analysis. Typical shortcomings of log file analysis include:

...