JobScheduler - Tutorial 5 - the JADE JITL Job I - editing and running a simple file transfer job

Introduction

JobScheduler has a full JADE managed file transfer program "on board". This allows JobScheduler features such as file polling and run time periods to be combined with JADE's managed file transfer features such as error detection, compliant logging and notifications.

JADE can be accessed from the JobScheduler using one of the JADE JITL jobs, via the JADE API. The JITL jobs are pre-configured job templates that can be used to rapidly develop job scheduling operations and are provided as standard with every JobScheduler installation package.

The configuration and use of a simple JITL job was described in JobScheduler - Tutorial 4 - Editing and Running a JITL Job: the current tutorial will build on the information provided in the Editing and Running a JITL Job tutorial and show how to configure, run and test simple file transfer operations. Users should then be able to build on this information to complete their own real-life transfer operations.

Background

The screenshots used in this tutorial were made using JobScheduler 1.9.x. All operations described have also been tested with 1.8.x. VERIFY

This tutorial assumes that a JobScheduler has been installed and is running. See the JobScheduler - Tutorials - Introduction and Requirements for an introduction to the requirements for the installation of JobScheduler and links instructions about the installation procedure itself.

This tutorial describes how to configure the JobScheduler JITL job to carry out a relatively simple file transfer operation - transferring all the files found in a folder on a remote server to the local file system. This operation described has already been used in the JADE file transfer tutorial, and involves the transfer of files from the online SOS demo server to the local file system. The demo server server allows read-only FTP and SFTP access and has been installed to allow users get their JADE and JADE JITL job users get their systems up, running and tested as quickly as possible.

Note that to get the file transfer operation described in this tutorial up and running, your firewall(s) will need to allow file transfer via ports 21 (for FTP) and 22 (for SFTP).

A separate installation of JADE is not required for this tutorial.

Parameter Settings in JADE

For ease of reference, the parameters specified in the JADE tutorial were:

Configuration of the JADE JITL job (FTP protocol, password authentication)

The first three steps have already been described in detail in the JobScheduler - Tutorial 2 - Editing a Simple Job with JOE tutorial and are therefore only summarized here:

1. Start JobScheduler
2. Open the JobScheduler Object Editor (JOE)
3. Open the JobScheduler Live Folder:
   1. A sub-folder of the live folder is used In this tutorial for all the objects related to the tutorial, "tutorials-5-jade-jitl-job".
      A link to a download file containing the contents of this file will be provided at the end of this article.
4. In the left hand pane of JOE mark the Jobs icon - opening the JobScheduler icon if necessary.
5. Start the Wizard function by clicking on the Button of the same name as shown in the screenshot below.
   
   
   
   
6. Select Order controlled job in the small window that opens and then Next, which will open the Import Jobs window as shown in the next screenshot.
   
   1. Enter a Jobname such as jitl_job_1.
   2. In the Jobs pane, scroll down until the two JADE-Jobs are visible and select the job with the jadeJob.xml file name.
      This will cause the Title field to be filled with "API Job for JobScheduler ...."
   3. Click on the Next button
      
      
      
      
7. The Job Parameter form for the JITL job will now open.
   
   • The use of this form to set and modify parameter values was described in detail in Tutorial 4 - Editing and Running a JITL Job.
     This form is shown in the screenshot below:
     
   • Note that some of the parameter names specified in the Wizard - e.g. local_dir - are similar to but not identical with those set in the JADE file transfer tutorial. This is because the JITL job Wizard still uses deprecated parameter settings without the source_ or target_ prefixes that were specified in the JADE job. This considerably restricts the configuration of the JITL Job and the general use of the Wizard to configure file transfer by directly selecting parameters is not recommended.
     Instead it is recommended that the Wizard is closed at this point - click on the Finish button - and that parameters are entered manually. There are a number methods where this can be done:
     • Use JOE'S job Parameter form to enter parameter name / value pairs. This is the simplest approach and will be used in this tutorial.
     • Specify a settings.ini file and profile in JOE'S job Parameter form and then enter the necessary parameters in this file using a text editor. This procedure will be described ..... TODO
     • Create a custom JITL job by duplicating the JADE JITL jjob in the JobScheduler data jobs folder and directly editing this file. This procedure will be described ..... TODO
8. Manually enter parameter name / value pairs in the Job Parameter form as shown in the next screenshot:
   
   
   
   • The Job Parameter form is opened by clicking on the Parameter icon for jitl_job_1 in the left hand pane of JOE.
   • Parameters are entered by simply typing names and values in the corresponding fields and then clicking the Apply button.
   • The screenshot shows all the parameters required to implement the simple JADE file transfer profile listed at the start of this tutorial.
     • The parameters shown are identical to those used in the JADE profile with the exception of the target_dir parameter which has been given an absolute value.
       This is because the JobScheduler on the system used to develop this tutorial does not run under the normal user Windows user account and therefore the ${USERPROFILE} variable would return the wrong path.
     • Information about parameter reference information will be given later on in this tutorial. TODO
9. Create a new job chain for the job:
   
   
   
   
10. Add the job to the job chain along with two end nodes as shown:
    
    
    
    
11. Create and configure an order for the job chain:
    
    
    
    
12. Don't forget to save the configuration using the dark grey button as shown:
    
    
    
    

13. Start the Order in JOC, the JobScheduler Operations Center using the Order menu button as shown
    
    

    
    
    
14. Open the log file after the order has run using the Show Log link as shown in the next screenshot:
    
    
    
    The screenshot below shows a extract from the log file for this order and the detailed log information that is available for JADE file transfer operations:
    
    
    
    As can be seen from the log file, six files have been copied to the ...\jade_demo\a folder. These were all the files in the remote folder as a filter was not specified to select particular files.

Further Configuration Topics

File Transfer using the SFTP protocol and public/private key authentication

JADE and therefore the JADE JITL Job can be configured to use a range of protocols, the most important of which is SFTP, and to use public/private key authentication.

The configuration of a working JITL job example using the SOS demo server is described in the JobScheduler - Tutorial 6 - the JADE JITL Job II - an SFTP job with public/private key authentication tutorial.

Use of a profile and settings file

The JADE JITL Job can also be configured to use file transfer configurations specified in the form of one or more profiles saved in a settings.ini file. This configuration method is used in th JADE Command Line Client up to Release 1.10 of JADE and JobScheduler and is described ... TODO

Error handling, notifications and monitoring

Error handling, notifications and monitoring for the JADE JITL job can be handled on two levels:

• using standard JobScheduler error handling, notifications and monitoring,
• using the JADE error handling and notifications

A brief introduction to both levels of approach is presented below.

JobScheduler error handling, notifications and monitoring

Error handling

The JobScheduler comes with a number of methods for error handling. These include:

• stop a job: running orders have to wait for the job to become available
• suspend an order: the order waits to be resumed later on
• setback an order: make an order repeatedly try to continue processing after a predefined time interval
• make an order leave a job chain
• make an order continue processing with a specific job node for error handling

See the Error handling article for a list of links to articles describing these features in detail.

Notifications

JobScheduler comes with its own mail client which it can use to send notifying e-mails in the event of, for example, jobs ending in error.

See the Notifications article for a list of links to articles describing the notification features in detail.

Monitoring

JobScheduler can be monitored by System Monitors. Such products include e.g. HP OpenView®, Microsoft SCOM®, Nagios®, op5®, Opsview®, Zabbix® etc.

As System Monitors are restricted to check the availability and performance of a monitored service the JobScheduler provides additional functionality for System Monitors to report on individual job failure and recovery. The JobScheduler Monitoring Interface can be used with any System Monitor that provides a command line tool for passive checks.

See the Monitoring article for a list of links to articles describing the monitoring features in detail.

JADE error handling and notifications

Error handling

JADE has a limited number of parameters that can be used to provide error handling:

• transactional and atomic file transfer
• check_size
• force_file
• raise_error_if_result_set_is

Definitions for these parameters are provided in the JITL Job Wizard Job Parameter form.

Notifications

Notification events

The JADE JITL job can send e-mails when file transfer events have taken place:

• on success (mail_on_success)
• on error (mail_on_error)
• on empty files (mail_on_empty_files)

Note that the exact definitions of these parameters are not available in the lower pane of the JITL Job Wizard Job Parameter form. RAISE ISSUE?

However the links behind the three links above lead to the relevant page of the JADE Parameter Reference which also contains valid descriptions of the functions of these parameters.

E-mail configuration

Notification e-mails themselves are configured in the JADE JITL job with a number of parameters whose (legacy) names begin with ftp_file_notification - e.g.  ftp_file_notification_to, ftp_file_notification_bcc, etc .

Exact definitions of these parameters can be found in the XML-based parameter reference under their new names - e.g. to, bcc, etc.

Scheduling the JADE JITL Job

Scheduling a JITL job in the JobScheduler is no different to scheduling any other job and therefore there is no reason to repeat the detailed instructions that are presented in JobScheduler - Tutorial 3 - Running an Order Using the Run Time Parameter

IST KORREKT?

Polling for File Presence

Two main methods for polling for the presence of a file are available:

• The JADE JITL Job can be used to poll as part of a file transfer operation. This is the simplest method to configure but binds the polling directly to the file transfer operation.
  • A configuration for carrying out file transfer from one server to another with polling is shown in the JADE Server-to-Server File Transfer with polling article.
    This configuration could be easily adapted for use with other file transfer situations.
• The JobScheduler can carry out the polling and provides a more powerful and flexible approach than the use of JADE. For example:
  • file order sources can be used to start a job chain that if required carries out a processing operation before the transfer itself. See the How to start jobs and job chains on a file event article for more information

Further documentation

JADE JITL job

As can be seen from the screen shot of the Job Parameter form above, documentation for individual JITL job parameters is shown in the lower pane of the Wizard Job Parameter form.

In addition, a full job documentation is provided alongside the job definition. This is an XML file (jadeJob.xml) and can be found in the JITL jobs folder in the JobScheduler installation Program Data/jobs directory. On a Windows 7/8 installation with default settings this would be:

• C:\ProgramData\sos-berlin.com\jobscheduler\scheduler_ID\jobs where scheduler_ID is the ID given to the JobScheduler. 

The full documentation for the JADE JITL job can also be found in the SOS Product Knowledge Base.

JADE parameter definitions

The lower pane of the JITL Job Wizard Job Parameter form (cut off in the screen shots above) shows an extract from the parameter documentation when a parameter that has been selected using the Wizard is active. This does not work when the parameter has been entered manually.

An alternative reference source, with in part more exact parameter descriptions can be found in the JADE Parameter Reference. Note that the version online has been written using the parameter names that will be introduced with the changeover to XML-based configuration in Release 1.11, and that we have withdrawn the legacy (i.e. settings.ini file based configuration) JADE parameter reference. However Release 1.11 does not bring a major change in the functions of the JADE parameters - only their names and how they are configured are significantly changed. Therefore we have made a Parameter Mapping available that shows the settings.ini file parameter names and their corresponding XML-based ones. The XML names can then be used to access the new JADE Parameter Reference, if there are aspects of parameter definitions that should be unclear.