Product Knowledge Base

Space shortcuts

Page tree

Introduction

JobScheduler has a full YADE managed file transfer program "on board". This allows JobScheduler features such as file watching and run time periods to be readily combined with YADE's managed file transfer features such as error detection, compliant logging and notifications.

YADE can be accessed from the JobScheduler using one of the YADE JITL jobs, via the YADE API. The JITL jobs are pre-configured job templates that can be used to rapidly develop job scheduling operations and are provided as standard with every JobScheduler installation package.

The configuration and use of a simple JITL job is described in JobScheduler - Tutorial 4 - Configuring and Running a JITL Job: the current tutorial builds on the information provided in that tutorial and shows how to configure and run and file transfer operations with the YADE Job. The current tutorial starts with the configuration of a simple file transfer operation and proceeds to showing how to include public/private key authentication and job configuration with external files. Users should then be able to use this information to configure their own real-life transfer operations.

List of Tutorials

The other tutorials in this series are:

Background

The screenshots used in this tutorial were made using JobScheduler 1.11.

This tutorial assumes that a JobScheduler has been installed and is running. See the JobScheduler - Tutorials - Introduction and Requirements for an introduction to the requirements for the installation of JobScheduler and links to instructions about the installation procedure itself.

This tutorial describes how to configure the YADE JITL Job to carry out a simple file transfer operation - transferring all the files found in a folder on a remote server to the local file system. This operation described has already been used in the The YADE Client Command Line Interface - Tutorial 1 - Getting Started, and involves the transfer of files from the online SOS test server to the user's local file system. The test server server allows read-only FTP and SFTP access and has been installed to help users get their jobs up, running and tested as quickly as possible.

Note that to get the file transfer operation described in this tutorial up and running, your firewall(s) will need to allow file transfer via ports 21 (for FTP) and 22 (for SFTP).

A separate installation of YADE is not required for this tutorial, the YADE JITL Job uses the JobScheduler's own YADE.

Example File Download

This tutorial describes the individual steps required to configure the Job, Job Chain and Order objects used in the examples. In addition, a completed configuration can be downloaded by users that just want to see how the examples work:

The download archive contains three sets of JobScheduler elements, each made up of a Job, a Job Chain and an Order. Each set of elements has a common suffix: _1, _2 or _3 and provides a working configuration for each of the three examples described in this tutorial:

  1. A simple JITL YADE Job (FTP protocol, password authentication),  
  2. File transfer using the SFTP protocol and public/private key authentication,
  3. Configuration using a Profile and settings.ini file.

The download archive can be unpacked and the contents copied to your JobScheduler live folder where they can be run with one minor change:

  • In the  settings.ini file (found in the tutorials-5-yade-jitl-job/yade_conf subfolder):
    • Change the username in the target_dir parameter from myUserName to an appropriate value for the user's system
      (Linux users will need to make wider changes to this parameter according to the requirements of their system.

Example 1 - Configuring and Running the YADE JITL Job (FTP transfer, password authentication)

Configuring the JITL Job in JOE

The first three steps have already been described in detail in the JobScheduler - Tutorial 2 - Editing a Simple Job with JOE tutorial and are therefore only summarized here:

  1. Start JobScheduler
  2. Open the JobScheduler Object Editor (JOE)
  3. In JOE, open the Open HotFolder function and use the Make New Folder to generate a sub-folder in the JobScheduler's Live folder. In this tutorial this sub-folder is given the name tutorials-5-yade-jitl-job. Now click OK to open this folder.
  4. In the left hand pane of JOE select the Jobs icon - opening the JobScheduler icon if necessary. Now use the wizard function to import the YADE JITL Job by clicking on the Wizard button as shown in the screenshot below.

    1. In the small window that will now open, select Order controlled job and then Next, which will open the Import Jobs window as shown in the next screenshot.
    2. Enter a Jobname such as yade_jitl_job_1, as shown in the screenshot.
    3. In the Jobs pane, scroll down until the two YADE Jobs are visible and select the second YADE-Job (the one without DMZ in its file name).
      This will cause the Title field to be filled with "API Job for JobScheduler ...." as shown in the next screenshot:

    4. Now click on the Finish button to create the new job and close the wizard.

      Deprecated Parameters

      Note that use of the wizard to configure the job parameters as described in Tutorial 4 - Configuring and Running a JITL Job  - i.e. proceeding by clicking on Next instead of Finish - is not recommended for this Job and an alternative method will be described.
      This is because in releases up to and including 1.9 the wizard uses parameters which have now been deprecated and which would considerably restrict the configurations possible for the JITL Job.
      For more information see JITL-201 - Getting issue details... STATUS .

    5. You will now be returned to JOE's Jobs tab where the yade_jitl_job_1 will be listed.
  5. Open the yade_jitl_job_1 in JOE's left hand pane and select the Parameter element as shown in the next screenshot.


    1. Now enter the parameter name/value pairs required to configure this Job.
      • There are a number of approaches that can be taken to specifying the parameter name/value pairs in for this Job:
        • Use the job Parameter form shown above to enter the parameter pairs and store the information in theJob configuration. This is the simplest approach and will be described first in this tutorial.
        • Specify a settings.ini file and Profile in job Parameter form and then enter the necessary parameters in this file using a text editor. This procedure is described in the Use of a Profile and settings.ini file section of this tutorial.
        • Note that it is also possible to create a custom JITL job by duplicating the YADE JITL job in the JobScheduler data jobs folder and directly editing this file. This procedure is outside the scope of this tutorial.

      • The values of the necessary parameter name/value pairs can be taken from the list below, which users that are familiar with the YADE tutorials will recognize as being the parameters for the first YADE tutorial:

        YADE Parameter Settings
         operation               = copy
 source_host             = test.sos-berlin.com
 source_protocol         = ftp
 source_port             = 21
 source_user             = demo
 source_password         = demo
 source_dir              = /
 target_host             = localhost
 target_protocol         = local
 target_dir              = ${USERPROFILE}\yade_demo\a

        Note that the  ${USERPROFILE} value must be replaced with a suitable path to the desired target directory.

      • The use of the job Parameter form is very similar to the use of the JITL job parameter wizard already mentioned. The use of the wizard was described in detail in the Configuring and Running a JITL Job tutorial and the use of the Parameter form should therefore not require any further explanation.
  6. The Job Parameter form will appear as shown below once all the parameter pairs have been entered:

    1. Note that the value of the target_dir parameter shown in the screenshot has been changed from the parametrized value used in YADE (i.e. ${USERPROFILE}\yade_demo\a) to an absolute value.
      This is because the JobScheduler on the system used to configure this tutorial runs under its own user account and therefore the ${USERPROFILE} variable would return the wrong path.
  7. Create a new Job Chain for the yade_jitl_job_1 job:


    1. Add the Job to the Job Chain along with two end nodes as shown:



    2. Create and configure an Order for the Job Chain:



    3. Don't forget to save the configuration as shown above.

Run the Order in the JOC Cockpit

The procedure for running this Order is the same as already described in other tutorials in this series:  

  1. Log into the JOC Cockpit and select the Job Chains view.
    In the left-hand tree menu select the tutorials-5-yade-jitl-job folder and in the yade_job_chain_1 card select Show Job Chain Details.



  2. Open the Order menu and select Start Order now.
    Processing of the Order will be shown in near real-time. Once processing has started the Job Chain card will be highlighted as shown in the next screenshot as well as in the Job Chain History list below:  


  3. After the Order has completed the status should show successful. Clicking on any entry in the History list will open the Order log file.
    The screenshot below shows an extract from the log file for this Order and the detailed log information that is available for YADE file transfer operations:



    As can be seen from the log file, six files have been copied to the ...\yade_demo\a folder. These were all the files in the remote folder as a filter was not specified to select particular files.

Troubleshooting

If an Order ends in error the Error Text can be conveniently read in the History view by clicking on the dark grey triangle to the left of the Order ID as shown in the next screenshot:

Firewall Problems 

Firewalls can cause problems and may block FTP file transfer if the necessary ports are not open.

In addition some firewalls require that FTP passive mode is used. An error message such as:

  • " Processing terminated, 'getFilenames' causes an exception error " 

is likely to be avoided if JOE is used to add a passive_mode parameter with value true to the parameters that were set for the yade_jitl_job_1 job as described above.

The default setting for the passive_mode parameter is false. 

Example 2 - File Transfer using the SFTP protocol and public/private key authentication

YADE and therefore the YADE JITL Job can be configured to use a range of protocols, including SFTP, and to use public/private key authentication. The configuration of the YADE JITL Job using SFTP and public/private key authentication is the subject of this section.

As with the simple file transfer using FTP protocol and password authentication described above, this job downloads files from the SOS demo server and saves them on the local file system. The configuration of the YADE Command Line Client to carry out this operation was described in The YADE Client Command Line Interface - Tutorial 6 - Public / Private Key Authentication tutorial and we recommend that you follow the instructions contained in that tutorial to download and install the necessary rsa private key file.

Job Configuration

  1. Public/private key installation:
    1. Follow the instructions in the YADE Tutorial 6 - Public / Private Key Authentication article (linked above) to download and correctly save the demo_id_rsa private key.
  2. Create a new Job, Job Chain and Order for this example:
    1. Either:
      1. Use the context menu for elements in JOE's left hand Scheduler Elements pane to copy and then paste the individual objects as shown in the next screenshot:
         
      2. or duplicate the objects at the file level and then load them into JOE using the 'Reload Configuration' icon (the arrow symbol with a grey background).
    2. The new Job Chain and Order objects now need to be modified so that the new Job Chain specifies the new Job and the new Order specifies the new Job Chain.
      See the JobScheduler - Tutorial 2 - Editing a Simple Job with JOE for instructions. 
    3. After the objects have been duplicated and modified the Scheduler Elements pane in JOE should appear similar the next screenshot:


    4. This initial configuration can now be tested by running the new Order in the JOC Cockpit using the Start Order Now function as described for Example 1 above. 
  3. The parameters in the yade_jitl_job_2 now need to be changed from FTP to SFTP and the authentication method from password to public key:
    1. The parameters required for these changes are shown in the next screenshot:
       
      1. The file_spec parameter has been added to introduce the YADE regex filter - instead of the seven files found and downloaded from the test server in Example 1 only one matching file will be found and downloaded - the test.txt file. 
      2. Once again, the use of the wizard is not recommended for modifying the parameters - instead they should be set using the editing functions provided directly in JOE's Parameters tab.
    2. Save the changes.
  4. Now run the Order using the JOC Cockpit in the same way that the Order for the yade_jitl_job_1 was run in Example 1 above.
    As already mentioned, the Order should cause one file to be downloaded from the test server to the local_dir folder - this file can be seen in the Order log file as shown in the next screenshot:  
     

Example 3 - Job Configuration using a Profile and settings.ini file

The YADE JITL Job can also be configured to use file transfer configurations saved in a file that is independent of the JITL job. This has the advantage that one instance of the configuration can be tested and reused in a number of JITL jobs.

Note that up to and including release 1.10 of YADE and JobScheduler both the YADE JITL job and the Command Line Client save configurations in the form of one or more Profiles in a text-based settings.ini file. From release 1.11 onwards this approach will be replaced with an XSD schema-based XML file. See the YADE User Manual - Using the YADE Client article for more information about this change.

Profiles can be called from other Profiles, which means that a system of reusable configuration fragments can be built up. This is described in the Example of a global profile with fragments article.

The profile and settings file are specified with parameters of the same names in the JITL job Parameter form, as shown in the screenshot below:

The screenshot also shows that a new Order, Job Chain and Job have been configured for this example. 

The full settings.ini file as included with the example download is listed in the following code block:

The settings.ini File
[ftp_server_2_local]
;;
  operation               = copy
;;
  source_host             = test.sos-berlin.com
  source_protocol         = ftp
  source_port             = 21
  source_user             = demo
  source_password         = demo
  source_dir              = /
;;
  target_host             = localhost
  target_protocol         = local
  target_dir              = C:\Users\myUserName\jade_demo\a
;;passive_mode            = true
 
[sftp_server_2_local]
;;
  operation               = copy
  file_spec               = ^test\.txt$
;;
  source_host             = test.sos-berlin.com
  source_protocol         = sftp
  source_port             = 22
  source_user             = demo
  source_ssh_auth_method  = publickey
  source_ssh_auth_file    = C:\Users\myUserName\jade_demo\demo_id_rsa
  source_dir              = /
;;
  target_host             = localhost
  target_protocol         = local
  target_dir              = C:\Users\myUserName\jade_demo\a
  • Note that in the above listing:
    • profile names are written between square brackets - e.g. [sftp_server_2_local],
    • the double semi-colons ( ";;" ) mark the start of comment lines.

The code list above shows that the settings.ini file contains profiles for both the FTP transfer operation described in the Example 1 and for the SFTP operation described in Example 2. Changing the value of the profile parameter in yade_jitl_job_3 will change the transfer operation carried out by the job.

Single Source Configuration

We recommend that only the settings and profile parameters are set in the JITL job parameter form and that all other configuration parameters are set in Profiles. For more information see JITL-202 - Getting issue details... STATUS

Note also:

  • The path to the settings.ini file is relative to the ${SCHEDULER_DATA} folder, which on a Windows computer with a default JobScheduler installation would be:
    • C:\ProgramData\sos-berlin.com\jobscheduler\JobScheduler_ID
      where JobScheduler_ID is the ID given to the JobScheduler during installation
  • We suggest that the settings.ini file is included in a folder within the JobScheduler live folder to simplify portability. For example:
    • settings = config\live\js-tutorials-yade-jitl-job\yade_conf\settings.ini
  • Parameter variables such as ${SCHEDULER_DATA}, which can be used when configuring a file transfer in the JITL Job Parameter form, are not interpreted from a settings file.
    • This means that the target_dir parameter which in the listing in the Parameter Settings in YADE subsection above has the value:
          ${USERPROFILE}\yade_demo\a
      is changed to:
          C:\Users\Alan\yade_demo\a
  • Parameter variables such as ${base_path} can be defined and used within a Profile. However they are not available in other profiles. For more information see  JADE-47 - Getting issue details... STATUS

Further Configuration Topics

Error handling, notifications and monitoring

Error handling, notifications and monitoring for the YADE JITL Job can be handled on two levels:

  • using standard JobScheduler error handling, notifications and monitoring,
  • using the YADE error handling and notifications

A brief introduction to both approaches is presented below.

JobScheduler Error Handling, Notifications and Monitoring

Error handling

The JobScheduler comes with a number of methods for error handling. These include:

  • stop a job: running orders have to wait for the job to become available
  • suspend an order: the order waits to be resumed later on
  • setback an order: make an order repeatedly try to continue processing after a predefined time interval
  • make an order leave a job chain
  • make an order continue processing with a specific job node for error handling

See the Error handling article for a list of links to articles describing these features in detail.

Notifications

JobScheduler comes with its own mail client which it can use to send notifying e-mails in the event of, for example, jobs ending in error.

See the Notifications article for a list of links to articles describing the notification features in detail.

Monitoring

JobScheduler can be monitored by System Monitors. Such products include e.g. HP OpenView®, Microsoft SCOM®, Nagios®, op5®, Opsview®, Zabbix® etc.

As System Monitors are restricted to check the availability and performance of a monitored service the JobScheduler provides additional functionality for System Monitors to report on individual job failure and recovery. The JobScheduler Monitoring Interface can be used with any System Monitor that provides a command line tool for passive checks.

See the Monitoring article for a list of links to articles describing the monitoring features in detail.

YADE Error Handling and Notifications

Error handling

YADE has a limited number of parameters that can be used to provide error handling:

  • transactional and atomic file transfer
  • check_size
  • force_file
  • raise_error_if_result_set_is

Definitions for these parameters are provided in the JITL Job Wizard Job Parameter form.

Notifications
Notification events

The YADE JITL job can send e-mails when file transfer events have taken place:

E-mail configuration

Notification e-mails themselves are configured in the YADE JITL Job with a number of parameters whose (legacy) names begin with ftp_file_notification - e.g.  ftp_file_notification_to, ftp_file_notification_bcc, etc .

Definitions of these parameters can be found in the Parameter Reference under their new names - e.g. to, bcc, etc.

Scheduling the JITL YADEJob

Scheduling a JITL job in the JobScheduler is no different to scheduling any other job, see JobScheduler - Tutorial 3 - Running an Order Using the Run Time Parameter

Polling for File Presence

Two main methods for polling for the presence of a file are available:

  • The YADE JITL Job can be used to poll as part of a file transfer operation. This is the simplest method to configure but binds the polling directly to the file transfer operation.
    • A configuration for carrying out file transfer from one server to another with polling is shown in the YADE Server-to-Server File Transfer with polling article.
      This configuration could be easily adapted for use with other file transfer situations.
  • The JobScheduler can carry out File Watching for directories within its reach. This provides a more powerful and flexible approach than the use of YADE or the YADE JITL Job.
    For example:
    • file order sources can be used to start a Job Chain. In addition to providing a more efficient and flexible, this approach processing operations to be carried out before the file transfer itself. See the How to start jobs and job chains on a file event article for more information.

Further Documentation

YADE JITL Job

A full job documentation of the YADE JITL Job is provided with the JobScheduler installation alongside the job definition. This documentation is an XML file and can be found in the JITL jobs folder in the JobScheduler installation Program Data/jobs directory. On a Windows installation with default settings this would be:

  • C:\ProgramData\sos-berlin.com\jobscheduler\scheduler_ID\jobs
    where scheduler_ID is the ID given to the JobScheduler. 

The full documentation for the YADE JITL Job can also be found in the JITL Jobs by topic article.

YADE parameter definitions

The lower pane of the JITL Job Wizard Job Parameter form (cut off in the screenshots above) shows an extract from the legacy and in part deprecated parameter documentation when a parameter that has been selected using the wizard is active. As already mentioned, we do not recommend using the wizard and we also do not recommend using its documentation.

An alternative reference source, with more reliable parameter descriptions can be found in the YADE Parameter Reference. Note that the version online has been written using the parameter names that will be introduced with the changeover to XML-based configuration in Release 1.11, and that we have withdrawn the legacy (i.e. settings.ini file based configuration) YADE parameter reference. However release 1.11 does not bring a major change in the functions of the YADE parameters - only their names and how they are configured is being significantly changed. Therefore we have made a Parameter Mapping available that shows the settings.ini file parameter names and their corresponding XML-based ones. The XML names can then be used to access the YADE Parameter Reference, if there are aspects of parameter use that is unclear.