Product Knowledge Base

Space shortcuts

Page tree

Versions Compared

Old Version 11

changes.mady.by.user Alan Amos

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Püschel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • The functions for terminating task processes by for the JobScheduler Master and Universal Agent have been extended to allow the use of both SIGTERM and SIGKILL signals on Unix servers:
    • SIGTERM is sent first and allows an orderly termination of task processes to take place within a limited period of time.
    • If the time allowed has been exceeded and the processes are still running then SIGKILL will be sent.
  • The information contained in this article draws together detailed information contained in a range of issues and should primarily be of interest to persons in engineering and to a lesser extent persons in operating functions.

Feature History

This feature has been implemented stepwise between releases 1.9.0 and 1.10.0 (see the table of issues below for more detailed information).

Display feature availability
StartingFromRelease1.9

Related Issues

Support of this feature is subject to the following issues:features:

  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1495
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1468
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution

...

  • serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1463
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1421
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1420
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1382
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1307
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1163
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution

...

  • serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJOC-10

Use Case

Users who schedule programs and scripts that are aware of SIGTERM signals can implement clean-up procedures on receipt of the signal. Clean-up includes e.g. removal of temporary files, disconnect from a database and similar tasks.

  • The use of both SIGTERM and SIGKILL signals on Unix servers has the following advantages:

    • The use of SIGTERM before SIGKILL means that there is a greater chance of data being saved after the signal has been issued.

    • The SIGTERM signal can - in contrast with SIGKILL - be monitored, i.e. a pre-/ post-processing Script can be carried out. This means that the ending of a task by the JobScheduler can be reacted to and the user process itself can be ended.

    • The implementation of SIGTERM allows post-processing methods such as spooler_process_after() to complete within the timeout period.

  • The time allowed between the SIGTERM and SIGKILL signals can be specified in the command using the timeout attribute (the default is 15 sec): <kill_task … timeout=".."/>

  • This feature can also be applied for:
    • remote processes, i.e. processes started by SSH and those started by an Agent,
    • child processes started by a process running on an agent (
      Jira
      serverSOS JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverId6dc67751-9d67-34cd-985b-194a8cdc9602
      keyJS-1468
      ).

Implementation

JobScheduler Master

...

  1. Operation: kill immediately
    • JOC sends <kill_task immediately="yes"/>
    • The process is killed immediately using the SIGKILL signal.
  2. Operation: terminate with timeout
    • JOC sends <kill_task immediately="yes" timeout="15"/>
    • The process receives a SIGTERM signal. Should that process not terminate within the specified timeout period then it will be killed with a SIGKILL signal.
  3. Operation: terminate
    • JOC sends <kill_task immediately="yes" timeout="never"/>
    • The process receives a SIGTERM signal. Monitoring of the process termination as described in Operation 2 above is not carried out.

See also

  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJOC-10

JobScheduler Universal Agent (JUA)

The JobScheduler Universal Agent comes with two scripts - for Windows and for Unix systems - that allow uses a script to monitor and kill parent and child tasks on the Agent to be killed.

The scripts are:

  • jobscheduler_agent_kill_task.cmd and
  • jobscheduler_agent_kill_task.sh

They are script is to be found  the ./bin folder of the JUA installation.

The script for the operating system being used is called with the parameter -kill-agent-task-id=...  and

  • finds the process containing the ID specified and
  • kills the process including all child processes.

The JUA start script starts the Agent with the new -kill-script parameter as follows:

  • by default the -kill-script is parameterized with the path to the kill script for Windows/Unix as appropriate,
  • the SCHEDULER_KILL_SCRIPT environment variable can be used to set a different kill script.

...

-kill-script option can be  used to specify a different kill script if necessary. See the Command Line Options section in JobScheduler Universal Agent - Installation & Operation for more information.

See also

  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1468
  • Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1495

for more detailed information about the implementation.

Delimitation

  • This feature is intended for Unix platforms that implement the SIGTERM and SIGKILL signals. It is not intended for Windows platforms for which exclusively the kill Immediately command applies.
  • When using traps then please consider that the process created by the <script ... language="shell"/> element receives the signal. Subsequent scripts that are called within the <script ... language="shell"/> element will not receive the signal. You could therefore:
    • configure traps directly within the <script ... language="shell"/> element. The shell process will then receive and handle the signal.
    • configure traps in a shell script that is added by an <include> element instead of being stated within the <script ... language="shell"/> element. The included shell script will receive and handle the signal.
    • forward signals to subsequent shell scripts that are called within a <script ... language="shell"/> element.
  • This feature has been fully implemented for the Universal Agent and for the Classic JobScheduler Agents (
    Jira
    serverSOS JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId6dc67751-9d67-34cd-985b-194a8cdc9602
    keyJS-1420
    ).

Implementation Summary

The implementation of the different termination operations available for the JobScheduler Master and Universal Agent is summarized in the table below.

The links lead to the issues relevant to the implementation of the feature.

The termination operations available are:

  • Terminate: <kill_task immediately="true" timeout="...">
  • Kill: <kill_task immediately="true">
  • Timeout: a job with the timeout attribute. StatuscolourYellowtitleVerify
JS-1420

JS-1421

server

JS-1468

serverId

JS-1468

key

JS-1463

key

JS-1463

serverId

JS-1468

JS-1468

server

JS-1382

JS-1468

key(tick)

JS-1382

 Windows
Standalone		Linux
Standalone		Windows
Classic Agent		Linux
Classic Agent		Windows
Universal Agent		Linux
Universal Agent
Shell job      
Terminatenot supported(tick)not supported

(tick)

(error) 2)

Jira
serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
keynot supported(tick)
Kill(tick)(tick)(tick)

(tick)jira

serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
key

(tick)

Jira
JIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
key

(tick)

Jira
serverJIRA Extern
6dc67751-9d67-34cd-985b-194a8cdc9602key
Timeout

(tick)

Jira
serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602

(tick)

Jira
serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
(tick)(tick) 

(tick)

Jira
serverJIRA Extern
6dc67751-9d67-34cd-985b-194a8cdc9602key
Shell job with monitor      
Terminatenot supported(tick)not supported(tick)not supported

(tick)

(error) 3)

Jira
serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
key
Kill(tick)(tick)(tick)(tick)

(tick)

Jira
JIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
key

(tick)

(error) 1)

Jira
serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
key
Timeout(tick)(tick)   

(error) 1)

Jira
serverJIRA Extern
serverId6dc67751-9d67-34cd-985b-194a8cdc9602
API job      
Terminatenot supported(tick)not supported(tick)not supported(tick)
Kill(tick)(tick)(tick)(tick)(tick)(tick)
Timeout(tick)(tick)    
  1. The child process continues: the shell job (and Java process if applicable) is killed, but the child process (sleep or ping - see attached jobs) is detached from the process tree and continues to run.
  2. No effect: neither the shell job nor it's children receive a signal
  3. The Java process is terminated but it's child process (the shell job script) and child processes thereof (sleep command) do not receive a SIGTERM signal
  4. 1.9.2-1.9.4 the kill comes too late, after the task has ended normally

Delimitation

  • This feature is intended for Unix platforms that implement the SIGTERM and SIGKILL signals. It is not intended for Windows platforms for which exclusively the Kill Immediately command applies.
  • When using traps then please consider that the process created by the <shell> element receives the signal. Subsequent scripts that are called within the <shell> element will not receive the signal. You could therefore:
    • configure traps directly within the <shell> element. The shell process will then receive and handle the signal.
    • configure traps in a shell script that is added by an <include> element instead of being stated within the <shell> element. The included shell script will receive and handle the signal.
    • forward signals to subsequent shell scripts that are called within a <shell> element.
  • This feature has been fully implemented on the Universal Agent and It has been implemented for classic JobScheduler Agents (JS-1420).

Workarounds

Workaround Title
Status
colourYellow
title???

  • Should job scripts not be able to catch signals by traps then you can use a monitor script, i.e. a post-processing script, that would be called by JobScheduler on receipt of a SIGTERM signal (JS-1463). For example:

    Code Block
    languagexml
    titleWorkaround for shell jobs with a timeout
    <job name="shell_with_javascript_monitor">
    <script  language="shell">
        <![CDATA[
echo hello world!
sleep 45
        ]]>
    </script>

    <monitor  name="process0" ordering="0">
        <script  language="java:javascript">
            <![CDATA[
function spooler_process_before(){
	return true;
}
            ]]>
        </script>
    </monitor>

    <run_time />
</job>

Examples

Task Termination in JOC

The following screenshot shows how a task can be terminated using the Task menu in JOC's Task pane. This pane is shown in the Jobs tab when the task is running and the task can be terminated either with timeout or immediately.

Image Added

Examples

Using a SIGTERM trap to show the difference between the kill and terminate commands

...

Download the Example

job_trap_sigterm.job.xml.zip

...

Code Block
languagexml
<?xml version="1.0" encoding="ISO-8859-1"?>
<job  title="test test">
    <script  language="shell">
        <![CDATA[
trap 'echo sigterm will be ignored' 15
for i in 1 2 3 4 5 6 7 8 9 0
do
date
sleep 10
done
sleep 60
        ]]>
    </script>

    <run_time />
</job>

Change Management References

Jira
serverSOS JIRA
columnstype,key,issuelinks,fixversions,status,priority,summary,updated
maximumIssues20
jqlQuerylabels in (terminate-tasks)
serverId6dc67751-9d67-34cd-985b-194a8cdc9602

References

...