Page History

Installing the JobScheduler Universal Agent

Prerequisites

• A Java Runtime Environment starting from version 1.8 is required.

Preparation

• Choose the JobScheduler Universal Agent archive for your platform the target system from the JobScheduler Downloads page.
  Installation
• Unzip the downloaded file to an arbitrary directory.
• Directory structure
  • bin
    • jobscheduler_agent.cmd
      • The start script for Windows platforms.
    • jobscheduler_agent.sh
      • The start script for Unix platforms.
  • lib
    • The directory for Java libraries.
    • Consider the settings in log4j.xml should you want to adjust the log output format.
  • var_4445
    
    • logs
    • config/private (for ssl configuration)
    • tmp (for internal use)
  • service (for Windows)
• On Windows Systems:
  • You have to change the directory permissions for the .\logs and .\service directories if you have extracted the JobScheduler Agent under e.g. C:\Program Files.
    • This step is not required if you extracted the JobScheduler Agent to e.g. C:\ProgamData.

    • Start a command prompt with elevated administrator rights and type e.g.

      Code Block
      language bash title set full acces for "Users" on .\service and .\logs directory
      cd "path\to\JobScheduler Agent installation" icacls "service" /L /grant *S-1-5-32-545:(OI)(CI)F icacls "logs" /L /grant *S-1-5-32-545:(OI)(CI)F

  • An installer is provided that is explained in detail with the JobScheduler Universal Agent - Installation with Windows Installer article.
    
    • The installer asks for elevated administrator privileges, therefore it is not necessary to change the permissions in the .\service directory and the installer sets read/write permissions for the log directory.
• On Unix systems:
  • A tarball is provided and a versatile batch installation feature that allows Agents to be deployed to hundreds of servers within minutes.
    • JobScheduler Universal Agent - Batch Installation and Update of Agents
  • Users requiring to install a small number of Agents on Unix should follow the instructions given in the JobScheduler Universal Agent - Running multiple instances article.
• For Unix systems:
  • A tarball archive is available that can be used for:
    • manual installation of one or more instances on a small number of computers.
      (The current article describes the installation of a single instance on a computer, the installation of multiple instances is described in the JobScheduler Universal Agent - Running multiple instances article.)
    • a versatile batch installation feature that allows Agents to be deployed to hundreds of servers within minutes.
      (Described in the JobScheduler Universal Agent - Batch Installation and Update of Agents article.)
• For Windows systems:
  • A .zip archive is available that can be used for:
    • manual installation on a small number of computers.
      (Described in the current article.)
    • the installation of a large number of Agents using third party deployment tools.
  • An installer that can be used for manual installation a small number of computers.
    (Described in detail in the JobScheduler Universal Agent - Installation with Windows Installer article.)

Installation

• • Unzip the downloaded file to an arbitrary directory.
  • Directory structure (only files and folders directly relevant ):
    • bin
      • jobscheduler_agent.cmd
        • The start script for Windows platforms.
      • jobscheduler_agent.sh
        • The start script for Unix platforms.
      • jobscheduler_agent_instance.sh-example
        • A template for configuring multiple instances of the Universal Agent
    • lib
      • The directory for Java libraries.
      • Configure the settings in log4j.xml file if you want to adjust the log output format.
    • var_4445
      
      • logs
      • config/private (for ssl configuration)
      • tmp (for internal use)
    • service (for Windows)
  • On Windows Systems:
    • You have to change the directory permissions for the .\logs and .\service directories if you have extracted the JobScheduler Agent under e.g. C:\Program Files.
      • This step is not required if you extracted the JobScheduler Agent to e.g. C:\ProgamData.

      • Start a command prompt with elevated administrator rights and type e.g.

        Code Block
        language bash title set full acces for "Users" on .\service and .\logs directory
        cd "path\to\JobScheduler Agent installation" icacls "service" /L /grant *S-1-5-32-545:(OI)(CI)F icacls "logs" /L /grant *S-1-5-32-545:(OI)(CI)F

  • If multiple instances are configured then every instance must have its own ./var_4445 data directory (e.g. ./var_<port of the instance>)

Update of a JobScheduler Universal Agent

• Preparations for an update
  
  • Stop all JobScheduler Universal Agent instances
  • Remove the ./lib directory
• Unzip the downloaded file to the installation directory
• If you use the Windows installer of JobScheduler Universal Agent then the ./lib directory is updated automatically during the setup. It is not necessary to remove the ./lib directory before you start the installer.

Running the JobScheduler Universal Agent

• SOS does not recommend to run running the JobScheduler Agent as root (for Unix) or as Administrator (for Windows).
• Instead the user account should be used that jobs are executed for. Should jobs be executed for a number of user accounts then consider the chapter Running multiple instances of JobScheduler Universal Agent.

...

Usage: jobscheduler_agent.sh command [options] [web services]
  command:
    start           [options]
    start_docker    [options]
    stop            [options]
    abort           [options]
    restart         [options]
    status          [options] [web services]
    start           [options]
 options:
    -http-port=<[hostname or ip address:]number>   | to listen to a specific host name or ip address if it is specified; default port: 4445, 
    -https-port=<[hostname or ip address:]number>  | 
    -data-directory=<location of data directory>   | default: ./var_4445
    -timeout=<number>                              | in seconds; only for stop and restart
    -kill-script=<location of kill script>         | default: "./bin/jobscheduler_agent_kill_task.sh"; only for start
    -java-options=<java options>                   | default: -Xms100m; see https://kb.sos-berlin.com/x/aIC9
  web services:
   -job-java-options=<java options>               | see https://kb.sos-berlin.com/x/aIC9
 web services:
    overwiew                                       | default
    task                                           | running JobScheduler Agent tasks
    tunnel                                         | tunnels summary
    tunnel/                                        | list of tunnels
    command                                        | running command summary
    command/                                       | list of running commands (watching file_order_source)

Usage: jobscheduler_agent.cmd command [options] [web services]
  command:
    start           [options]
    stop            [options]
    abort           [options]
    restart         [options]
    status          [options] [web services]
    debug           [options]
    kill            [options]
    install-service [options]
    remove-service  [options]
    start-service   [options]
 options:
    -http-port=<[hostname or ip address:]number>   | to listen to a specific host name or ip address if it is specified; default port: 4445, 
    -https-port=<[hostname or ip address:]number>  | 
    -data-directory=<location of data directory>   | default: ./var_4445
    -timeout=<number>                      | in seconds; only for stop and restart
    -kill-script=<location of kill script> | default: ".\bin\jobscheduler_agent_kill_task.cmd"; only for start, debug
    -java-options=<java options>           | default: -Xms100m; see https://kb.sos-berlin.com/x/aIC9
  web services:
   -job-java-options=<java options>  overwiew     | see https://kb.sos-berlin.com/x/aIC9
 web services:
    overwiew                               | default
    task                                   | running JobScheduler Agent tasks
    tunnel                                 | tunnels summary
    tunnel/                                | list of tunnels
    command                                | running command summary
    command/                               | list of running commands (watching file_order_source)

...

• -http-port=<[hostname or ip address:]number>
  •  is the HTTP port that the Agent is listening to in order to receive requests from a JobScheduler Master:

    • jobscheduler_agent.cmd|sh command -http-port=####

    • where #### is the numeric port.
    • This option can be also used for indicating which network interfaces the JobScheduler Agent should listen to if you specify a hostname or ip address in addition
  • Without this option being used the port defaults to 4445 and the JobScheduler Agent listens to all available network interfaces.
  • Should you want to specify a port then the following precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable SCHEDULER_HTTP_PORT (see below)
    • Third precedence: use of default value
• -https-port=<[hostname or ip address:]number>
  •  is the HTTPS port that the Agent is listening to in order to receive requests from a JobScheduler Master:

    • jobscheduler_agent.cmd|sh command -https-port=####

    • where #### is the numeric port.
    • This option can be also used for indicating which network interfaces the JobScheduler Agent should listen to if you specify a hostname or ip address in addition
  • Without this option being used the port defaults to 4445and the JobScheduler Agent listens to all available network interfaces.
  • Should you want to specify a port then the following precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable SCHEDULER_HTTP_PORT (see below)
    • Third precedence: use of default value
• -data-directory=<number>
  • Location of the data directory.
  • It has to be unique over all JobScheduler Universal Agent instances
  • Should you want to specify a data directory then the following precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable SCHEDULER_DATA (see below)
    • Third precedence: use of default value (=SCHEDULER_HOME/var_SCHEDULER_HTTP_PORT)
• -timeout=<number>
  • This option can be used to specify the number of seconds that the Agent will wait for tasks to stop. 
  • This option can be applied for stop and restart commands.
  • The Agent sends a SIGTERM signal to the taksk and having reached the timeout a SIGKILL signal will be sent to stop any tasks immediately.
• -kill-script=<location of kill script>

  • The kill scripts provide the functionality to kill a task and it's child processes.

  • Two kill scripts are provided
    

    • ./bin/jobscheduler_agent_kill_task.sh for Unix as default

    • .\bin\jobscheduler_agent_kill_task.cmd for Windows as default

  • This option can be used to specify the location of a different "kill script" if necessary

  • Should you want to specify a different "kill script" then the following precedence applies:

    • First precedence: command line option
    • Second precedence: environment variable SCHEDULER_KILL_SCRIPT (see below)
    • Third precedence: use of default value
• -java-options=<java options>

  • With Java 1.8 the initial memory allocation has changed, for details see How to manage the Java heap space.

  • This option can be used to apply Java options for the JobScheduler Universal Agent, e.g. the memory settings.

  • Without this option being used the Java options default to '-Xms100m'.

  • Should you want to specify the Java options then the following precedence applies:

    • First precedence: command line option
    • Second precedence: environment variable JAVA_OPTIONS (see below)
    • Third precedence: use of default value

Start the Agent

• • • value
• -job-java-options=<java options>

  • Without this option being used the Java options for each job which is started by the JobScheduler Universal Agent.

  • Should you want to specify the Java options for the jobs then the following precedence applies:

    • First precedence: command line option
    • Second precedence: environment variable SCHEDULER_JOB_JAVA_OPTIONS (see below)

Start the Agent

jobscheduler_agent.cmd

jobscheduler_agent.cmd|sh start [options]

Stop the Agent

...

• The Agent waits for running processes to be completed.
• Any child processes will be killed by the Agent.

...

jobscheduler_agent.cmd|sh abort [options]

...

• Any running tasks and child processes are killed immediately with a SIGKILL signal.
• Should task have used resources such as database connections then they will not be properly closed.

jobscheduler_agent.cmd|sh kill [options]

...

isTerminating: false
system:
  hostname: agenthost
  mxBeans:
    operatingSystem:
      processCpuLoad: 2.5630713121704744E-5
      availableProcessors: 4
      freePhysicalMemorySize: 311668736
      systemCpuLoad: 0.046373903924522855
      committedVirtualMemorySize: 4475301888
      totalPhysicalMemorySize: 3155517440
java:
  systemProperties:
    java.vendor: Oracle Corporation
    os.arch: amd64
    os.version: 2.6.32-220.17.1.el6.x86_64
    os.name: Linux
    java.version: 1.8.0_31
version: 1.10.0-SNAPSHOT (6956c56a535d15fcf659f293c42d22dcf92e9e12 2015-07-15 21:23:24+02:00)
startedAt: '2015-07-17T08:38:30.516Z'
totalTaskCount: 21
currentTaskCount: 0

Should the Agent not be running then some output is provided such as:

ERROR: spray.can.Http$ConnectionAttemptFailedException: Connection attempt to localhost:4445 failed
...JobScheduler Agent(4445) not started!

Monitoring Agents

See the For an automated status check of Agents for monitoring purposes consider the information from the article: How to perform active checks with a System Monitor such as Nagios/op5 article for information about configuring an automated status check of Agents for monitoring purposes.

Master/Agent Compatibility

...

After the installation of the Windows Service you will find the .\service\sos_jobscheduler_agent_<http-port>w.exe file. Start this program to configure the Windows Service.

Image Modified

For example goto the "Startup" tab to modifiy the start parameter

Change Management References - Windows Service

Anchor
logging logging

Logging

...

• JAVA_HOME
  • points to the location of the Java Runtime Environment (JRE).
  • Without setting this environment variable Java will be used from the location specified by the system path.
  • Please consider that JAVA_HOME does not point to the location of a JDK but to a JRE directory where the bin/java executable resides.
• JAVA_OPTIONS
  • sets Java options, e.g the Java memory settings for the JobScheduler Universal Agent.
  • Without setting this environment variable the Java options defaults to '-Xms100m'.
• SCHEDULER_HOMEJOB_JAVA_OPTIONS
  • sets Java options for each job which is started by the JobScheduler Universal Agent.
• SCHEDULER_HOME
  
  • points to the
  
  
  • points to the directory where the JobScheduler Agent has been installed.
  • Without setting this environment variable the default value is the parent directory of the start script.
  • Should you want to start the Agent from a directory different to the Agent installation directory, e.g. by copying the start script to some other location, then this environment variable has to be set in order to locate the JobScheduler Agent installation directory.
• SCHEDULER_DATA
  • points to the directory where the JobScheduler Agent has its data directory.
  • Without setting this environment variable the default value is SCHEDULER_HOME/var_SCHEDULER_HTTP_PORT.
• SCHEDULER_HTTP_PORT
  • sets the http port that the JobScheduler Agent is listening to.
  • indicates which network interfaces the JobScheduler Agent should listen to if a host or ip address specified.
  • if only a port number is specified then the JobScheduler Agent listens to all available network interfaces via http.
  • Without setting this environment variable the port defaults to 4445.
• SCHEDULER_HTTPS_PORT
  • sets the https port that the JobScheduler Agent is listening to.
  • indicates which network interfaces the JobScheduler Agent should listen to if a host or ip address specified.
  • if only a port number is specified then the JobScheduler Agent listens to all available network interfaces via https.
  • Without setting this environment variable the https protocol doesn't use.
• SCHEDULER_USER
  • sets the user account that the JobScheduler Agent is operated for. This includes running jobs with the permissions of the specified user.
  • This setting is available for Unix systems only. For Windows systems the user account that runs the start script is used.
  • Without setting this environment variable the user acount that runs the start script is used.
  • This setting can be used when running the Agent start script in system start-up and shutdown configurations that are executed by root, e.g. in /etc/init.d or corresponding locations.
• SCHEDULER_LOG_DIR
  • sets the directory where the JobScheduler Agent log file is created.
  • This setting defaults to the directory logs in the Agent installation directory.
  • For Windows systems for which the Agent is installed in the program directory that is pointed to by the %ProgramFiles% environment variable it is recommended not to use the default setting. Instead specify a different path via the SCHEDULER_LOG_DIR environment variable, e.g. some location in the data directory that is pointed to by the %ProgramData% environment variable.
• SCHEDULER_WORK_DIR
  • sets the working directory for the jobs started by the JobScheduler Agent, e.g. ${HOME} or %USERPROFILE%.
  • This setting defaults to the SCHEDULER_HOME.
• SCHEDULER_KILL_SCRIPT
  • sets the location of a "kill script" if necessary.

  • The kill scripts provide the functionality to kill a task and it's child processes.

  • Two kill scripts are provided
    

    • ./bin/jobscheduler_agent_kill_task.sh for Unix as default

    • .\bin\jobscheduler_agent_kill_task.cmd for Windows as default

• SCHEDULER_PID_FILE_DIR
  • sets the directory where the JobScheduler Agent pid file is created.
  • This setting defaults to the directory that is specified with the SCHEDULER_LOG_DIR environment variable or the log directory default value.

...

• For Unix systems the start-up and shutdown configurations apply that are executed by root, e.g. in /etc/init.d or corresponding locations.
  • Consider use of the SCHEDULER_USER environment variable to run an Agent that is started by root for a different user account.
• For Windows systems the start-up of the Agent by installing it as a Windows Service is recommended.

...

Debugging

• The Agent log level can be increased using the Agent's Apache ProcRun Demon Service Manager demon/service.
• On Windows systems this is installed in the Agent service Folder and will have a name such as sos_jobscheduler_agent_4444w.exe where 4444 is the port the agent can be addressed over. 
• Start the ProcRun Manager, select the Logging tab in the Manager interface and set the level to Debug. 
• The location of the log files has already been described above. 
• (Do not forget to set the debug level back to Info once finished.)

...

Anchor
running_multiple_instances running_multiple_instances

Running multiple instances of JobScheduler Universal Agent

• Multiple instances of the JobScheduler Agent on the same computer can be operated, e.g. for different user accounts that jobs should be executed for.
• Consider detailed instructions with the JobScheduler Universal Agent - Running multiple instances article.

...

set JAVA_HOME=%ProgramFiles%\Java\jre8
 
"%ProgramFiles%\sos-berlin.com\agent\jobscheduler_agent_1.10\bin\jobscheduler_agent.cmd" start -http-port=4446

set JAVA_HOME=%ProgramFiles%\Java\jre8
set SCHEDULER_LOG_DIR=%ProgramData%\sos-berlin.com\agent\jobscheduler_agent_1.10\logs
set SCHEDULER_HTTP_PORT=4446
 
"%ProgramFiles%\sos-berlin.com\agent\jobscheduler_agent_1.10\bin\jobscheduler_agent.cmd" start

su - js
jobscheduler_agent.sh start -http-port=4446

SCHEDULER_USER=js
SCHEDULER_HTTP_PORT=4446
export SCHEDULER_USER SCHEDULER_HTTP_PORT

jobscheduler_agent.sh start

Testing the JobScheduler Universal Agent Operability

A simple way to test if the installed JobScheduler Agent works as expected is to carry out a test using Process Classes. The Process Class and the corresponding job will be defined in your JobScheduler Master, which will connect to the JobScheduler Agent. SOS recommends to do the test locally to exclude connection problems, e.g. firewall settings, since the goal of the test is to see whether the JobScheduler Agent is working properly.

Process Class and Standalone Job

First create a Process Class, for example the following (find attached example: agent1.process_class.xml):

...

• http://, e.g. like for the above http://localhost:4445, for HTTP communication.
• https:// if you did configure secure HTTPS communication.

Second, you should define a Standalone Job and associate the Process Class as configured above to the job (find an example attached: standalone_hello.job.xml):

...

Find more details and use cases from the article: How to execute Jobs and Job Chains with Agents

Logs

The easiest way to check if the test was successful is to verify the contents of the job log in JOC and to see if the job completed successfully. Otherwise an error will be reported in JOC.

...

2015-04-28 16:11:24.462 +0200 [DEBUG] akka.io.TcpListener - New connection accepted
......
......
2015-04-28 16:13:39.231 +0200 [DEBUG] spray.can.server.HttpServerConnection - TcpConnection terminated, stopping

Remote File Watching

] spray.can.server.HttpServerConnection - TcpConnection terminated, stopping

Remote File Watching

To configure remote file watching, see JobScheduler Universal Agent - Remote File Watching

Behavior in the event of the Universal Agent crashing

It is important that all the tasks running on a Universal Agent are killed if the Agent should crash or otherwise terminate abnormally while executing tasks. To this end, every task that is being executed by a Universal Agent is noted by a jobscheduler_agent_watchdog.sh agent watchdog process in a kill_tasks_after_crash.sh script. This script is located in the Agent's tmp folder and tasks are dynamically added to and deleted from the script as they are started and completed. This script is dynamically created when a first task is started by the agent and deleted when no tasks are running.

If a Universal Agent crashes the Agent Watchdog will start this script which will then cause all tasks that were running when the Agent crashed to be killed.To configure remote file watching, see JobScheduler Universal Agent - Remote File Watching

See also

• Installation
  • JobScheduler Universal Agent - Batch Installation and Update of Agents
  • JobScheduler Universal Agent - Installation with Windows Installer
  • JobScheduler Universal Agent - Running multiple instances
• Operation
  • How to execute Jobs and Job Chains with Agents
  • JobScheduler Universal Agent - Remote File Watching

...