Introduction / Scope

A batch install and update solution is available for the JobScheduler Universal Agent which allows installation of any number of agents on multiple servers.

This article describes the use of this solution. 

Requirements and Initial Configuration

The Agent Batch Install/Update can be downloaded from the attached files and comes with a job chain and the AgentBatchInstaller job for the JobScheduler that will be used for the batch operation.

You also need a JobScheduler Agent installation file. The current release can be downloaded here: https://www.sos-berlin.com/jobscheduler-downloads

The AgentBatchInstaller job reads an XML configuration file provided with the batch install download and uses it to create orders for the job chain that will install/update the Agents. There is an entry in the installer configuration file for each Agent.

The parameters for the job chain steps are configured with the AgentBatchInstaller job.

FEATURE AVAILABILITY STARTING FROM RELEASE 1.10

The JobScheduler Universal Agent Batch Installer is available with  JITL-214 - Getting issue details... STATUS

Universal Agent Batch Installer Job Chain

Job Chain: universal_agent_installer.job_chain.xml

The job chain makes use of the following configuration:

<job_chain  name="universal_agent_installer">
    <job_chain_node  state="CheckPreRequisite" job="CheckPreRequisite" next_state="Sync" error_state="error"/>
    <job_chain_node  state="Sync" job="Sync" next_state="TransferInstallationSetup" error_state="error"/>
    <job_chain_node  state="TransferInstallationSetup" job="TransferInstallationSetup" next_state="PerformInstall" error_state="error"/>
    <job_chain_node  state="PerformInstall" job="PerformInstall" next_state="success" error_state="error" on_error="suspend"/>
    <job_chain_node  state="success"/>
    <job_chain_node  state="error"/>
</job_chain>

The job chain for update/installation of an Agent uses the nodes:

• CheckPreRequisite
  • Checking for the correct Java version
  • Checking whether JobScheduler Universal Agent already has been installed.
    
    • If so
      • shutting down a possibly running instance
      • remove the installation
• Sync: To wait until all prerequisites are checked and all instances have been shut down.
• TransferInstallationSetup: To transfer the installer files to the host
  • Before the transfer starts, the preprocessor will create the instance start script from the template jobscheduler_agent_instance_script.txt

  • The files that will be transfered are specified with the configuration element <globals><transfer><file_spec=".*\.(sh|gz)$">

  • With this regular expression, the following files will be transfered:
    
    • the JobScheduler Universal Agent installer files: jobscheduler_unix_universal_agent.1.11.0-SNAPSHOT.tar.gz
    • the instance start script: jobscheduler_agent_<port>.sh
    • the installation script: jobscheduler_universal_agent_install.sh
• PerformInstall: To start the installation using SSH.
  • The step before transfered installation script will be executed
  • The installation script will install and start the JobScheduler Universal Agent
  • Finally the postprocessing commands will be executed
    • The postprocessing commands are stored with the element <installation><postprocessing><command>
    • Three commands will be executed
      
      • if you have more than three commands, please add the call in the job PerformInstall
    • The default commands will setup the autostart for JobScheduler Universal Agent on a Linux server (Ubuntu distribution)
      
      • echo ${ssh.sudo_password} | sudo -S cp ${install_path}/bin/jobscheduler_agent.sh /etc/init.d/jobscheduler
      • echo ${ssh.sudo_password} | sudo -S update-rc.d jobscheduler defaults

Job Parameters

All jobs have access to the following parameters. These parameters are used for the JobScheduler Universal Agent instance start script.

<job chain>_requird_orders

agent_options.scheduler_ip_address

agent_options.scheduler_http_port

agent_options.java_home

agent_options.java_options

agent_options.scheduler_home

agent_options.scheduler_user

agent_options.scheduler_log_dir

agent_options.scheduler_kill_script

agent_options.scheduler_pid_file_dir

host

port

user

auth_method

auth_file

password

sudo_password

install_path

Job: TransferInstallationSetup

More detailed explanations of the parameters are available with the YADE - Documentation.

It is recommended that the YADE settings file is specified with the <globals> element and that the default profile name <host>:<port> is used. The <installation><transfer> element is not then required for the installer configuration file.

settings

profile

file_spec

target_host

target_port

target_protocol

target_user

target_password

target_dir

target_ssh_auth_method

target_ssh_auth_file

source_host

source_port

source

source_user

source_password

source_dir

source_ssh_auth_method

source_ssh_auth_file

Job: PerformInstall

More detailed explanations of the parameters are available with the JITL SSH job documentation

command_n

command_counter

Starting the update/installation

The update/installation procedure will be started by running the job AgentBatchInstaller.

The Job AgentBatchInstaller

The job AgentBatchInstaller reads the installer configuration file and creates orders for each element <installation>.

The element <installation> defines the source and the target of the installer files, the port for the JobScheduler Universal Agent and the SSH connection parameters. Please note:

• For all elements you can define defaults in the <globals> element.
• You can use variables for the values e.g. ${ssh.host}.
• It is recommended to specify the file name of the YADE settings file in the <globals> element and to use the default profile name. When doing so, no <installation><transfer> elements are required.

Job Parameters

installation_definition_file

filter_install_host

filter_install_port

installation_job_chain

update

Customizing the Installation Process

You can customize the installation process.

• Changing the script part in the SSH jobs CheckPreRequisite and PerformInstall e.g. if you want to check other prerequisites or execute some commands before or after the installation.
• Changing the installation script jobscheduler_universal_agent_install.sh
• Changing the template for the instance start script jobscheduler_agent_instance_script.txt

Checking Prerequisites

The job CheckPreRequisite will check the Java version, shutdown a possibly running JobScheduler Universal Agent and remove the lib folder from an existing installation.

echo .JobScheduler Universal Agent
echo .checking prerequisites ${install_path}
echo .checking Java Runtime Version >= 1.8? ......
JAVA_VER=$(java -version 2>&1 | sed 's/java version "\(.*\)\.\(.*\)\..*"/\1\2/; 1q')
if [ "$JAVA_VER" > "18" ]
then
  echo ..java version: $JAVA_VER ok
else
  echo ..wrong Java version found
  echo ..expectet version is >= 1.8
  echo ..installed version is $JAVA_VER
  exit 8
fi
echo .checking whether JobScheduler Universal Agent is already installed?
if [ -f ${install_path}/jobscheduler_agent/bin/jobscheduler_agent.sh ]
then
  echo ..JobScheduler Universal Agent already installed
  echo ..shutting down JobScheduler Universal Agent 
  ${install_path}/jobscheduler_agent/bin/jobscheduler_agent.sh stop
  echo ..removing installation from  ${install_path}
  rm -Rf ${install_path}/lib
else
  echo .. this is the first installation of JobScheduler Universal Agent 
fi
exit $?

Performing the Installation

The job PerformInstall calls the installation script and executes the commands specified with the <postprocessing> element.

echo Executing PerformInstall ${target_dir}
echo ${sudo_passwort} | sudo -S chmod a+x ${target_dir}/jobscheduler_universal_agent_install.sh 
echo ${sudo_passwort} | sudo -S ${target_dir}/jobscheduler_universal_agent_install.sh -d${install_path} -p${agent_options.scheduler_http_port}
${command_0}
${command_1}
${command_2}

The Installation Script

The installation script will unzip the installation file and start the JobScheduler Universal Agent.

#!/bin/sh 
# Copyright (c) 2015 SOS GmbH, Berlin, Germany.
# All rights reserved.
export TERM=dumb
  clear
  echo "+---------------------------------------------------------+"
  echo "*     JobScheduler Univeral Agent                         *"
  echo "*     SOS Software- und Organisations Service GmbH        *" 
  echo "*     Giesebrechtstr. 15                                  *"
  echo "*     10629 Berlin                                        *"
  echo "*     ----------------------------------------------      *"
  echo "*     info@sos-berlin.com                                 *"
  echo "*     http://www.sos-berlin.com                           *"
  echo "+---------------------------------------------------------+"
  echo ""
  echo ""
  echo "+---------------------------------------------------------+"
  echo "*  This will install JobScheduler Universal Agent now ... *"
  echo "+---------------------------------------------------------+"
 
# -----------------------------------------------
cd `dirname $0`
  
SETUP_WORKING_DIR=`pwd` 
SETUP_INSTALL_PATH=""
SHOW_EXTRACTED_FILESNAMES=0
SETUP_LOG_FILE="$SETUP_WORKING_DIR/jobscheduler_universal_install.log"
SETUP_ERR=0
 


# ----------------------------------------------------------------- 
log_write() {
  case "$1" in
      0  ) msg_hint="[info]   ";;
      1  ) msg_hint="[warning]";;
      2  ) msg_hint="[error]  ";;
      3  ) msg_hint="[fatal]  ";;
      *  ) msg_hint="[info]   ";;
  esac
  timestamp=`date "+%Y-%m-%d %T"`
  msg="$timestamp   $msg_hint   $2"
  echo "$msg"
  echo "$msg">>"$SETUP_LOG_FILE"
}

show_usage() {
  echo "`basename $1` will install the 'JobScheduler Universal Agent' at [install_dir]." 
  echo "install_dir=Path with the JobScheduler Universal Agent"
  echo
  echo "Usage: `basename $1` [OPTION]"
  echo
  echo "Samples:"
  echo "  `basename $1` -co  -d/home/jobscheduler  -p4445 "      
  echo 
  echo "Options:"
  echo "  -h, --help                          | Shows this usage"
  echo "  -s, --show                          | show all extracted filenames"
  echo "  -w  --install_dir                   | Path of the JobScheduler Universal Agent installation"
  echo "  -p  --port                          | Port of JobScheduler Universal Agent. Default=4445"
 
}
 
# ----------------------------------------------------------------------------usage
if [ $# -eq 0 ]
then
  show_usage $0
  exit 65
fi 



# --------------------------------------------
     

for arg in "$@"
do
  long_opt=0
  echo "`expr match $arg '\(..\)'`" 
  case "`expr match $arg '\(..\)'`" in
    "-h"        )  show_usage $0; exit 64;;
    "-d"        )  SETUP_INSTALL_PATH="`expr match $arg '-d\(.*\)'`";;
    "-p"        )  UNIVERSAL_AGENT_PORT="`expr match $arg '-p\(.*\)'`";;
    "-s"        )  SHOW_EXTRACTED_FILESNAMES="`expr match $arg '-d\(.*\)'`";;
    "--"        )  long_opt=1;;
  esac 
  
  if [ "$long_opt" -eq 1 ]
  then
  case `expr match "$arg" '\(--[^=]*\)'` in
    "--show_extracted_filenames"    )  SHOW_EXTRACTED_FILESNAMES=`expr match "$arg" '--.*=\(.*\)'`;;
    "--help"    )  show_usage $0; exit 64;;
    "--install_dir"    )  SETUP_INSTALL_PATH=`expr match "$arg" '--.*=\(.*\)'`;;
    "--port"    )  UNIVERSAL_AGENT_PORT=`expr match "$arg" '--.*=\(.*\)'`;;
  esac
  fi

done

 
# ---------------

  
check_dir(){
     SETUP_INSTALL_PATH="$1"
     echo SETUP_INSTALL_PATH="$SETUP_INSTALL_PATH"

     if [ -d "$SETUP_INSTALL_PATH" ]
     then
       if [ -r "$SETUP_INSTALL_PATH" ]
       then
         cd "$SETUP_INSTALL_PATH"
         SETUP_INSTALL_PATH=`pwd`
         chmod 700 "$SETUP_INSTALL_PATH" 2>/dev/null
         SETUP_ERR=$?
       else
         log_write 2 "$SETUP_INSTALL_PATH is not readable."
         SETUP_ERR=1
       fi
     else
       echo create $SETUP_INSTALL_PATH

       mkdir -m700 -p "$SETUP_INSTALL_PATH" 2>/dev/null
       SETUP_ERR=$?
       if [ $SETUP_ERR -eq 0 ]
       then
      
         cd  "$SETUP_INSTALL_PATH"
         SETUP_INSTALL_PATH=`pwd`
       else
         log_write 2 "$SETUP_INSTALL_PATH can not create."
         SETUP_ERR=1 
       fi
     fi
     
     
     cd  "$SETUP_WORKING_DIR"
     if [ $SETUP_ERR -ne 0 ]
     then
        log_write 2 "You haven't the necessary rights on the installation directory $SETUP_INSTALL_PATH."
     exit $SETUP_ERR
fi
}

tarx (){
  if [ -f "$1.gz" ]
  then
     gzip -fd $1.gz
  fi
  
  if [ ! -d "$2" ]
  then
      mkdir $2
  fi
 
  cd  $2
  tar $tarpar $SETUP_WORKING_DIR/$1
  cd  $SETUP_WORKING_DIR
} 

tarpar="-xf "

if [ "$SHOW_EXTRACTED_FILESNAMES" = "1" ]
then
   tarpar="-xvf "
fi


SETUP_INSTALL_PATH="$SETUP_INSTALL_PATH"

echo SETUP_INSTALL_PATH=$SETUP_INSTALL_PATH
check_dir $SETUP_INSTALL_PATH
  

# Installing
log_write 0 "Installing --> $SETUP_INSTALL_PATH"
tarx  jobscheduler_unix_universal_agent.tar  $SETUP_INSTALL_PATH 

if [ -z "$UNIVERSAL_AGENT_PORT" ]
then
$UNIVERSAL_AGENT_PORT=4445
fi

mv `dirname $0`/jobscheduler_agent_$UNIVERSAL_AGENT_PORT.sh $SETUP_INSTALL_PATH/jobscheduler_agent/bin 
chmod a+x  $SETUP_INSTALL_PATH/jobscheduler_agent/bin/jobscheduler_agent_$UNIVERSAL_AGENT_PORT.sh   
log_write 0 "Start JobScheduler Universal Agent"
log_write 0 $SETUP_INSTALL_PATH/jobscheduler_agent/bin/jobscheduler_agent_$UNIVERSAL_AGENT_PORT.sh start
$SETUP_INSTALL_PATH/jobscheduler_agent/bin/jobscheduler_agent_$UNIVERSAL_AGENT_PORT.sh start 1 > /dev/null

exit $?

The Installer Configuration File

The installer configuration file contains an <installation> element for each JobScheduler Universal Agent instance.

A minimal example for the installer configuration file can look like this:

 <?xml version="1.0" encoding="utf-8"?>
<installations lastRun="2015-11-06 15:11">
  <globals>
    <transfer>
      <settings><![CDATA[C:\Temp\batch_install\yade_agent_batch_installer_settings.ini]]></settings>
    </transfer>
  </globals>
  <installation lastRun="2015-11-06 15:11">
    <install_path><![CDATA[/home/test/temp/xxx]]></install_path>
    <agent_options>
      <scheduler_http_port>4445</scheduler_http_port>
      <scheduler_ip_address><![CDATA[192.11.0.116]]></scheduler_ip_address>
    </agent_options>
    <ssh>
      <user><![CDATA[test]]></user>
      <password><![CDATA[12345]]></password>
      <sudo_password><![CDATA[12345]]></sudo_password>
      <auth_method><![CDATA[password]]></auth_method>
      <port><![CDATA[22]]></port>
    </ssh>
    <postprocessing>
      <command><![CDATA[echo ${ssh.sudo_password} | sudo -S cp ${install_path}/bin/jobscheduler_agent.sh /etc/init.d/jobscheduler]]></command>
      <command><![CDATA[echo ${ssh.sudo_password} | sudo -S update-rc.d jobscheduler defaults]]></command>
    </postprocessing>
  </installation>
</installations>

With the <globals> element you specify the values that are used for all installation items. The items that are specific for an individual installation will overwrite the global values. There is one installation item for each Agent. You can use substitution in the values of the elements.

<command>echo ${ssh.sudo_password} | sudo cp ${install_path}/bin/jobscheduler_agent.sh /etc/init.d/jobscheduler</command>

Using the XML Editor

Then XML Editor can be used to manage the installater configuration file. It can be used to manage the YADE profiles for the transfer of the installation files as well.

Find more details from our XML Editor section.

• After download of the XML Editor, extract the zip file, then start "sos.xml.editor.exe"
• Open your installer configuration file and assign the XSD schema scheduler_universal_agent_installations.xsd

Managing the YADE profiles for the file transfer

Each installation makes use of a profile in the YADE settings file. The default profile name is <host>:<port> where

• <host> will be substituted with the value from the element <installation><agent_option><scheduler_ip_address>
• <port> will be substituted with the value from the element <installation><agent_option><scheduler_http_port>

1. Specify the name of the YADE settings file with the <globals><settings> element.
2. Open the YADE settings file or create a new one.
3. Assign the XSD Schema http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd
4. Add a <ProtocolFragments> element and nodes for each host that you want to install the JobScheduler Universal Agent for.
5. Add a <Profile> element and nodes for each host you want install the JobScheduler Universal Agent for.
6. The default profile_id is <host>:<port>. If you do not use the default then you have to specify the profile name in the installer configuration file.

Installation of the Batch Installer for the JobScheduler Universal Agent

Please follow the following steps:

• Download the JobScheduler Universal Agent Batch Installer zip File
  
  • jobscheduler_universal_agent_batch_install.zip
• Unzip the JobScheduler Universal Agent Batch Installer zip file.
• Download the JobScheduler Universal Agent installation file.
  • https://www.sos-berlin.com/jobscheduler-downloads

For Unix:

• • Download the "JobScheduler Universal Agent" tarball for Unix.
  • Download the "JobScheduler Universal Agent Batch Installer" for Unix. The archive contains the following: 
    • a config/live/batch_installer_agent folder, with the jobs and job chains that are to be extracted to the live folder of the JobScheduler Master: SCHEDULER_DATA/config/live.
    • a config/batch_installer folder containing configuration files for the Batch Installer. These files should be extracted to the config directory of the JobScheduler Master: SCHEDULER_DATA/config.
  • Copy the JobScheduler Universal Agent installation file to the SCHEDULER_DATA/config/batch_installer folder.

For Windows:

• • Download the "JobScheduler Universal Agent" for Windows.
  • Download the "JobScheduler Universal Agent Batch Installer" for Windows. The archive provides the following content:
    • the config/live/batch_installer_agent folder contains jobs and job chains that should be extracted to the live folder of the JobScheduler Master SCHEDULER_DATA/config/live.
    • the config/batch_installer folder contains configuration files for the Batch Installer that should be extracted to the JobScheduler configuration directory SCHEDULER_DATA/config.
    • Copy the JobScheduler Universal Agent installation file to the folder ${SCHEDULER_DATA}/config/batch_installer.

Now the following:

• Rename the JobScheduler Universal Agent installation file to jobscheduler_unix_universal_agent.tar.gz
• Create the installer configuration file batch_installer_config_file.xml
• Adjust the parameter values for the JobScheduler job AgentBatchInstaller
  • Adjust the installation_definition_file job parameter with the path that points to the Batch Installer configuration file.Set the name for the YADE settings file in the installer configuration file with the element <globals><transfer><settings>.
• Create the YADE settings file 
  • Add a <ProtocolFragments> element and nodes for each host that you want to install the JobScheduler Universal Agent for.
  • Add a <Profile> element and nodes for each host that you want to install the JobScheduler Universal Agent for.