Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 85 Next »

Installing the JobScheduler Universal Agent

  • Prerequisites
    • A Java Runtime Environment starting from version 1.8 is required.
    • Choose the JobScheduler Universal Agent for your platform from the JobScheduler Downloads.
  • Installation
    • Unzip the downloaded file to an arbitrary directory.
    • Directory structure
      • bin
        • jobscheduler_agent.cmd
          • The start script for Windows platforms.
        • jobscheduler_agent.sh
          • The start script for Unix platforms.
      • lib
        • The directory for Java libraries.
        • Consider the settings in logback.xml should you want to adjust the log output format.
      • logs
      • service (for Windows)
    • On Windows you have to change the directory permissions for the .\logs and .\service directories if you have extracted the JobScheduler Agent under e.g. C:\Program Files.
      • This step is not required if you extracted the JobScheduler Agent to e.g. C:\ProgamData.

      • Start a command prompt with elevated administrator rights and type e.g.

        set full acces for "Users" on .\service and .\logs directory
        cd "path\to\JobScheduler Agent installation"
icacls "service" /L /grant *S-1-5-32-545:(OI)(CI)F
icacls "logs" /L /grant *S-1-5-32-545:(OI)(CI)F 
    • For Windows an installer is provided that is explained in detail with the JobScheduler Universal Agent - Installation with Windows Installer article.
      • The installer asks for elevated administrator privileges, therefore it is not required to change the permissions in the .\service directory and the installer sets read/write permissions for the log directory.
    • For Unix a tarball is provided and a versatile batch installation feature that allows to deploy Agents to hundreds of servers within minutes.

Running the JobScheduler Universal Agent

  • SOS does not recommend to run the JobScheduler Agent as root (for Unix) or as Administrator (for Windows).
  • Instead the user account should be used that jobs are executed for. Should jobs be executed for a number of user accounts then consider the chapter Running multiple instances of JobScheduler Universal Agent.

Usage

Running the start script without parameters shows the usage clause:

usage on Unix
Usage: jobscheduler_agent.sh command [options] [web services]
  command:
    start           [options]
    start_docker    [options]
    stop            [options]
    abort           [options]
    restart         [options]
    status          [options] [web services]
    start           [options]
 options:
    -http-port=<number>                    | default: 4445
    -timeout=<number>                      | in seconds; only for stop and restart
    -ip-address=<hostname or ip address>   | to listen to a specific host name or ip address
    -kill-script=<location of kill script> | default: "./bin/jobscheduler_agent_kill_task.sh"; only for start
    -java-options=<java options>           | default: -Xms100m; see https://kb.sos-berlin.com/x/aIC9
  web services:
    overwiew                               | default
    task                                   | running JobScheduler Agent tasks
    tunnel                                 | tunnels summary
    tunnel/                                | list of tunnels
    command                                | running command summary
    command/                               | list of running commands (watching file_order_source)
usage on Windows
Usage: jobscheduler_agent.cmd command [options] [web services]
  command:
    start           [options]
    stop            [options]
    abort           [options]
    restart         [options]
    status          [options] [web services]
    debug           [options]
    kill            [options]
    install-service [options]
    remove-service  [options]
    start-service   [options]
 options:
    -http-port=<number>                    | default: 4445
    -timeout=<number>                      | in seconds; only for stop and restart
    -ip-address=<hostname or ip address>   | to listen to a specific host name or ip address
    -kill-script=<location of kill script> | default: ".\bin\jobscheduler_agent_kill_task.cmd"; only for start, debug
    -java-options=<java options>           | default: -Xms100m; see https://kb.sos-berlin.com/x/aIC9
  web services:
    overwiew                               | default
    task                                   | running JobScheduler Agent tasks
    tunnel                                 | tunnels summary
    tunnel/                                | list of tunnels
    command                                | running command summary
    command/                               | list of running commands (watching file_order_source)

Command Line Options

  • -http-port=<number>
    •  is the HTTP port that the Agent is listening to in order to receive requests from a JobScheduler Master:
      • jobscheduler_agent.cmd|sh command -http-port=####
      • where #### is the numeric port.
    • Without this option being used the port defaults to 4445.
    • Should you want to specify a port then the following precedence applies:
      • First precedence: command line option
      • Second precedence: environment variable SCHEDULER_HTTP_PORT (see below)
      • Third precedence: use of default value
  • -timeout=<number>
    • This option can be used to specify the number of seconds that the Agent will wait for tasks to stop. 
    • This option can be applied for stop and restart commands.
    • The Agent sends a SIGTERM signal to the taksk and having reached the timeout a SIGKILL signal will be sent to stop any tasks immediately.
  • -ip-address=<hostname or ip address>

    • This option can be used for indicating which network interfaces the JobScheduler Agent should listen to.

    • Should you want to specify an ip address then the following precedence applies:
      • First precedence: command line option
      • Second precedence: environment variable SCHEDULER_IP_ADDRESS (see below)
    • Without this option being used and the environment variable SCHEDULER_IP_ADDRESS is undefined then the JobScheduler Agent listens to all available network interfaces.
  • -kill-script=<location of kill script>

    • The kill scripts provide the functionality to kill a task and it's child processes.

    • Two kill scripts are provided

      • ./bin/jobscheduler_agent_kill_task.sh for Unix as default

      • .\bin\jobscheduler_agent_kill_task.cmd for Windows as default

    • This option can be used to specify the location of a different "kill script" if necessary

    • Should you want to specify a different "kill script" then the following precedence applies:

      • First precedence: command line option
      • Second precedence: environment variable SCHEDULER_KILL_SCRIPT (see below)
      • Third precedence: use of default value
  • -java-options=<java options>

    • With Java 1.8 the initial memory allocation has changed, for details see How to manage the Java heap space.

    • This option can be used to apply Java options, e.g. the memory settings.

    • Without this option being used the Java options default to '-Xms100m'.

    • Should you want to specify the Java options then the following precedence applies:

      • First precedence: command line option
      • Second precedence: environment variable JAVA_OPTIONS (see below)
      • Third precedence: use of default value

Start the Agent

jobscheduler_agent.cmd|sh start [options]

Stop the Agent

jobscheduler_agent.cmd|sh stop [options]

This command will safely terminate the Agent (recommended).

  • The Agent waits for running processes to be completed.
  • Any child processes will be killed by the Agent.

 

jobscheduler_agent.cmd|sh abort [options]

The Agent process is terminated immediately.

  • Any running tasks and child processes are killed immediately with a SIGKILL signal.
  • Should task have used resources such as database connections then they will not be properly closed.

 

jobscheduler_agent.cmd|sh kill [options]

The Agent process is killed.

  • This corresponds to sending SIGKILL with a kill command.
  • Should jobs be running that started detached child processes then it is not guaranteed that child processes will be killed.

Restart the Agent

jobscheduler_agent.cmd|sh restart [options]

Query the Agent Status

jobscheduler_agent.cmd|sh status [options]

Should the Agent be up and running then this command will result in some output such as:

isTerminating: false
system:
  hostname: agenthost
  mxBeans:
    operatingSystem:
      processCpuLoad: 2.5630713121704744E-5
      availableProcessors: 4
      freePhysicalMemorySize: 311668736
      systemCpuLoad: 0.046373903924522855
      committedVirtualMemorySize: 4475301888
      totalPhysicalMemorySize: 3155517440
java:
  systemProperties:
    java.vendor: Oracle Corporation
    os.arch: amd64
    os.version: 2.6.32-220.17.1.el6.x86_64
    os.name: Linux
    java.version: 1.8.0_31
version: 1.10.0-SNAPSHOT (6956c56a535d15fcf659f293c42d22dcf92e9e12 2015-07-15 21:23:24+02:00)
startedAt: '2015-07-17T08:38:30.516Z'
totalTaskCount: 21
currentTaskCount: 0

 

Should the Agent not be running then some output is provided such as:

ERROR: spray.can.Http$ConnectionAttemptFailedException: Connection attempt to localhost:4445 failed
...JobScheduler Agent(4445) not started!

 

For an automated status check of Agents for monitoring purposes consider the information from the article: How to perform active checks with a System Monitor such as Nagios/op5

Windows Service Interface: Usage

The following information applies to batch installation on Windows systems. For installation with a GUI and user dialog see JobScheduler Universal Agent - Installation with Windows Installer.

The JobScheduler Universal Agent is operable as a Windows Service. The start script of the JobScheduler Universal Agent allows to install/remove the Windows Service:

Install the Windows Service

jobscheduler_agent.cmd install-service [-http-port=<number>] [-ip-address=<hostname or ip address>]

This command installs the Windows Service. After the installation you find the Windows Service with the name SOS JobScheduler Agent -port=<number> in the Services Manager Console. The Windows service uses the "local system" account.

During the service installation it tries to copy the executable file for the Windows Service to the .\service directory. This operation could fail with the error "Access denied" if you have extracted the JobScheduler Universal Agent to e.g. C:\Program Files\. In this case you can change the permissions of the .\service directory or open the command prompt with elevated administrator rights and execute the above command once more.

Start the Windows Service

jobscheduler_agent.cmd start-service [-http-port=<number>]

This command starts the Windows Service with the name SOS JobScheduler Agent -port=<number>.

The stop command contains more than a simple stop-service command: the stop command checks whether the Agent was started through the CLI or as a Windows Service and stops the Agent accordingly. Therefore there is no stop-service command.

Remove the Windows Service

jobscheduler_agent.cmd remove-service [-http-port=<number>]

This command removes the Windows Service. After executing this command you will not find the Windows Service with the name SOS JobScheduler Agent -port=<number> in the Services Manager Console any longer.

During removal of the service it tries to remove the executable file of the Windows Service from the .\service directory. This operation could fail with the error "Access denied" if you have extracted the JobScheduler Agent to e.g. C:\Program Files\. In this case you can change the permissions of the .\service directory or open the command prompt with elevated administrator rights and execute the above command once more.

Configure the Windows Service

After the installation of the Windows Service you will find the .\service\sos_jobscheduler_agent_<http-port>w.exe file. Start this program to configure the Windows Service.

For example goto the "Startup" tab
to modifiy the start parameter

 

Logging

  • Log File
    • On startup the Agent creates a log file in the directory that is pointed to by the environment variable SCHEDULER_LOG_DIR or in the logs subdirectory of the Agent installation directory.
    • Log file names are created from a prefix and the port used by the Agent like this:
      • jobscheduler_agent_4445.log
    • Log files are rotated for each day (see ./lib/log4j.xml) for which job activities occur.
    • Rotated log files are assigned file names like this: jobscheduler_agent_4445.log.<yyyy-MM-dd>
    • For days where the JobScheduler Agent has no jobs to execute no log rotation will be performed.
  • PID File
    • On startup the Agent creates a PID file in the directory that is pointed to by the environment variable SCHEDULER_PID_FILE_DIR or in the log directory. The PID file contains the Process ID of the system process that the Agent is running in.
    • The PID file is used in order to prevent the Agent to be started twice with the same settings and it can be used for shutdown scripts that require the PID to terminate the process.
    • PID file names are created from a prefix and from the port used by the Agent like this:
      • jobscheduler_agent_4445.pid

Environment Variables

The following environment variables can be used:

  • JAVA_HOME
    • points to the location of the Java Runtime Environment (JRE).
    • Without setting this environment variable Java will be used from the location specified by the system path.
    • Please consider that JAVA_HOME does not point to the location of a JDK but to a JRE directory where the bin/java executable resides.
  • JAVA_OPTIONS
    • sets Java options, e.g the Java memory settings.
    • Without setting this environment variable the Java options defaults to '-Xms100m'.
  • SCHEDULER_HOME
    • points to the directory where the JobScheduler Agent has been installed.
    • Without setting this environment variable the default value is the parent directory of the start script.
    • Should you want to start the Agent from a directory different to the Agent installation directory, e.g. by copying the start script to some other location, then this environment variable has to be set in order to locate the JobScheduler Agent installation directory.
  • SCHEDULER_HTTP_PORT
    • sets the port that the JobScheduler Agent is listening to.
    • Without setting this environment variable the port defaults to 4445.
  • SCHEDULER_IP_ADDRESS

    • indicates which network interfaces the JobScheduler Agent should listen to.

    • The JobScheduler Agent listens to all available network interfaces if this environment variable and the command line option -ip-address are undefined.
  • SCHEDULER_USER
    • sets the user account that the JobScheduler Agent is operated for. This includes running jobs with the permissions of the specified user.
    • This setting is available for Unix systems only. For Windows systems the user account that runs the start script is used.
    • Without setting this environment variable the user acount that runs the start script is used.
    • This setting can be used when running the Agent start script in system start-up and shutdown configurations that are executed by root, e.g. in /etc/init.d or corresponding locations.
  • SCHEDULER_LOG_DIR
    • sets the directory where the JobScheduler Agent log file is created.
    • This setting defaults to the directory logs in the Agent installation directory.
    • For Windows systems for which the Agent is installed in the program directory that is pointed to by the %ProgramFiles% environment variable it is recommended not to use the default setting. Instead specify a different path via the SCHEDULER_LOG_DIR environment variable, e.g. some location in the data directory that is pointed to by the %ProgramData% environment variable.
  • SCHEDULER_WORK_DIR
    • sets the working directory for the jobs started by the JobScheduler Agent, e.g. ${HOME} or %USERPROFILE%.
    • This setting defaults to the SCHEDULER_HOME.
  • SCHEDULER_KILL_SCRIPT
    • sets the location of a "kill script" if necessary.

    • The kill scripts provide the functionality to kill a task and it's child processes.

    • Two kill scripts are provided

      • ./bin/jobscheduler_agent_kill_task.sh for Unix as default

      • .\bin\jobscheduler_agent_kill_task.cmd for Windows as default

  • SCHEDULER_PID_FILE_DIR
    • sets the directory where the JobScheduler Agent pid file is created.
    • This setting defaults to the directory that is specified with the SCHEDULER_LOG_DIR environment variable or the log directory default value.

Usage Examples

Running the JobScheduler Agent on Windows

For Windows® operating systems the location of the Java Runtime Environment and of the log directory can be specified like this:

set JAVA_HOME=%ProgramFiles%\Java\jre8
set SCHEDULER_LOG_DIR=%ProgramData%\sos-berlin.com\agent\jobscheduler_agent_1.10\logs 
"%ProgramFiles%\sos-berlin.com\agent\jobscheduler_agent_1.10\bin\jobscheduler_agent.cmd" start

Running the JobScheduler Agent on Mac OS X

For Mac® OS X the location of the Java Runtime Environment can be specified like this:

JAVA_HOME=/Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home
export JAVA_HOME
/Users/ap/Documents/jobscheduler_agent/bin/jobscheduler_agent.sh start

Automated Start-up and Shutdown of JobScheduler Universal Agent

  • For Unix systems the start-up and shutdown configurations apply that are executed by root, e.g. in /etc/init.d or corresponding locations.
    • Consider use of the SCHEDULER_USER environment variable to run an Agent that is started by root for a different user account.
  • For Windows systems the start-up of the Agent by installing it as a Windows Service is recommended.

Running multiple instances of JobScheduler Universal Agent

Testing the JobScheduler Universal Agent Operability

A simple way to test if the installed JobScheduler Agent works as expected is to carry out a test using Process Classes. The Process Class and the corresponding job will be defined in your JobScheduler Master, which will connect to the JobScheduler Agent. SOS recommends to do the test locally to exclude connection problems, e.g. firewall settings, since the goal of the test is to see whether the JobScheduler Agent is working properly.

Process Class and Standalone Job

First create a Process Class, for example the following (find attached example: agent1.process_class.xml):

Example Process Class
<?xml version="1.0" encoding="ISO-8859-1"?>
<process_class  max_processes="10">
    <remote_schedulers >
        <remote_scheduler  remote_scheduler="http://localhost:4445"/>
    </remote_schedulers>
</process_class>

Important! When defining the remote JobScheduler URL, it is required to add 

 

Second, you should define a Standalone Job and associate the Process Class as configured above to the job (find an example attached: standalone_hello.job.xml):

Example Standalone Job
<?xml version="1.0" encoding="ISO-8859-1"?>

<job  process_class="agent1">
    <script  language="shell">
        <![CDATA[
echo "hello world"
        ]]>
    </script>
    <run_time />
</job>

Finally:

  • store both XML files (agent1.process_class.xml and  standalone_hello.job.xml) to the JobScheduler live folder in ./config/live.
  • start the job in JOC (start task immediately). 

Find more details and use cases from the article: How to execute Jobs and Job Chains with Agents

Logs

The easiest way to check if the test was successful is to verify the contents of the job log in JOC and to see if the job completed successfully. Otherwise an error will be reported in JOC.

For double-checking you can verify the log from the JobScheduler Universal Agent. If the JobScheduler Universal Agent is running and the steps above were correctly completed then you should see something like this in the JobScheduler Universal Agent log:

JobScheduler Universal Agent Log
2015-04-28 16:11:24.462 +0200 [DEBUG] akka.io.TcpListener - New connection accepted
......
......
2015-04-28 16:13:39.231 +0200 [DEBUG] spray.can.server.HttpServerConnection - TcpConnection terminated, stopping

Remote File Watching

To configure remote file watching, see JobScheduler Universal Agent - Remote File Watching

See also

 

  • No labels