Connection Keep-Alive for Master and Agent

Scope

• JobScheduler Master and Classic and Universal Agents can maintain network connections by regularly sending keep-alive packets. This prevents connections across firewalls being broken if a long-running job should exceed allowed timeouts of firewalls or proxy servers.
• FEATURE AVAILABILITY STARTING FROM RELEASE 1.9

Related Issues

JS-1454 - Getting issue details... STATUS

JS-1456 - Getting issue details... STATUS

JS-1506 - Getting issue details... STATUS

Use Case

• Increasing awareness of security issues means that an increasing number of firewalls and proxy servers are to be found within company intranets, where they, for example, can form part of the infrastructure used to create so-called network islands.
• This means that the interception of idle network connections between JobSchedulers - in particular between JobScheduler Masters and Agents - by firewalls or proxies can start to represent a significant problem.
• Idle connections occur when, for example, a long-running job is being processed by an Agent and the job does not create log output from time to time, meaning that data is not sent over the network.
• The feature described here uses so-called keep-alive packets that are sent across network connections at regular intervals.
• If interception takes place then this can happen either politely or impolitely:
  • Polite interception means that the firewall sends a connection reset packet to the JobScheduler Master.
    
    • This means that the Master is able to rebuild the connection.
  • Impolite interception means that the firewall simply drops subsequent packets and that information from an Agent about, for example, job completion will be lost.

Implementation

JobScheduler Master

• The JobScheduler Master can be configured to send an empty command to the JobScheduler Agent to prevent connections from timing out by adding a scheduler.agent.keep_alive parameter to the <params> section of the Master's scheduler.xml file. This file is located in the $SCHEDULER_DATA/config folder, where $SCHEDULER_DATA is the directory used for JobScheduler's configuration and log files.

  keep-alive parameter

  <params>
      <param name="scheduler.agent.keep_alive" value="300"/>
  </params>

• The value attribute sets the interval in seconds between keep-alive packets.

  Keep-alive packets will not be sent if the parameter is not set or if the value attribute is empty.

• The Master sends keep-alive commands to Classic Agents (including and before Agent release 1.9) via TCP connections. VERIFY
• The Master log will show a SCHEDULER-727 Keep-alive package sent to Agent message at the info level stating that a keep-alive command has been successfully sent.
  • A duration lower than 30s is silently replaced by 30s.

JobScheduler Classic Agent

• The JobScheduler Classic Agent can also be configured to send a small keep-alive packet to the JobScheduler Master to prevent connections from timing out by adding the following parameter to the <params> section of the Agent's scheduler.xml file.

  keep-alive parameter

  <params>
      <param name="scheduler.agent.keep_alive" value="300"/>
  </params>

• The value attribute sets the interval in seconds between keep-alive packets.

  Keep-alive packets will not be sent if the parameter is not set or if the value attribute is empty.

• The Master log will show a SCHEDULER-727 Keep-alive package sent to Agent message at the info level stating that a keep-alive command has been successfully sent. VERIFY
  • A duration lower than 30s is silently replaced by 30s.

Master / Universal Agent Feature Compatibility

• Master and Universal Agent use only one connection (HTTP, HTTPS): keep-alive packets do not have to be sent by the Master to the Universal Agent as the Agent (starting with release 1.10) sends keep-alive packets itself - see JS-1456. VERIFY

Workarounds for Older Versions

The idle connection can also be overcome by:

• regularly calling API methods (only in jobs using the JobScheduler's API),
• regularly logging to stdout (every log line to stdout is a log API call as well and makes use of the TCP connection used by the Classic Agent to communicate with the JobScheduler Master), VERIFY
• using one of the heartbeat scripts provided below.

Heartbeat script for Linux Agents

• Store the following script heartbeat.sh in the Agent's file system:

  Heartbeat script for Linux Agents

  trap 'kill $(jobs -p)' EXIT
  /path/to/heartbeat.sh &
  
  #your scriptcode here...

• Configure long running job scripts to run the heartbeat script.

Heartbeat script for Windows Agents

• Store the following heartbeat.cmd script in the Agent's file system:

  Heartbeat script for Windows Agents

  start /b C:\temp\heartbeat.cmd
  			
  @rem your scriptcode here...

• Configure long-running job scripts to run the heartbeat script.
• Don't use quotes in the start command, e.g. start /b "c:\my scripts\heartbeat.cmd".

Heartbeat script download

• heartbeat_scripts.zip

Delimitation

• Keep-alive packets are not created if Remote File Watching is performed by the Agent.
• Keep-alive packets are only sent for running jobs.

Change Management References

References

• Change Management References

  • JIRA Issues

• Documentation

  • XML Element in der Referenzdokumentation