Product Knowledge Base

Space shortcuts

Page tree

Introduction

The YADE Background Service provides automatic centralized collection and storage of file transfer records as well as giving operators a detailed ongoing record of completed file transfer operations.

The Background Service can collect file transfer records from any number of YADE Clients and store this information in a database. The History Viewer web interface is available for examining ongoing file transfer records and provides comprehensive filtering for sorting records. In addition notifications can be sent to operators and monitoring systems.

Architecture

YADE comes with two main components: the Client, which is responsible for the file transfer itself and the YADE History, which shows the file transfer history in JOC.

Components

The YADE Client is made up of a number of sub-components:

YADE Client

    • Seamless integration with JobScheduler for automated transfer

    • Adds entries to the YADE History

YADE History

The history is shown in JOC.

About this tutorial

  • This tutorial uses the simple FTP file transfer example described in the The YADE Client Command Line Interface - Tutorial 1 - Getting Started article as a basis for demonstrating the configuration of the Background Service.
  • In addition, the use of the XML Editor to generate the settings.ini file used to specify the file transfer and Background Service configurations is described. This editor generates both the settings.ini configuration file required for YADE versions up to and including 1.10 and the XML configuration file required for version 1.11 onwards as well as validating the input against an XSD schema before generating the configuration files. This will considerably simplify the configuration process and reduce development and testing requirements. 
  • Links behind parameter names are to the parameter reference article for that parameter.

Other YADE tutorials

See the YADE - Tutorials article for an overview of other YADE tutorials.

Requirements

In addition to the YADE Command Line Client, this tutorial requires that a JobScheduler and YADE Background Service are installed. In addition, we strongly recommend that the XML Editor is used to configure the Background Service:

Background Service Information Retrieval Modes

The YADE Background Service Automation Jobs import file transfer records generated after each file transfer operation has been completed in two independent but complimentary modes:

  • Immediate mode:
    • After completion of a file transfer operation the YADE Client sends detailed history records using the UDP or TCP protocols to a port that is being monitored by the JobScheduler providing the Background Service.
      • The UDP protocol operates in a "fire and forget" manner - i.e. without feedback - so that the YADE Client is not aware whether the history file has been received by the Background Service.
      • Note that we strongly recommend that the UDP protocol is used here and not TCP as this has the advantage that errors will not occur in the event of a network connection not being able to be made.
  • Batch mode:
    • This is provided by the YADE Background Service Automation Jobs - a job chain run by the JobScheduler providing the Background Service. This job chain reads a persistently stored CSV history file in which the YADE Client writes file transfer records after each transfer operation has been completed. The Background Service Automation job chain then saves this information to the Background Service database.

Intended Usage:

We have designed the two Background Service modes to be used together to provide quick and reliable information about the operations that have been completed by YADE:

  • Immediate mode transfer per UDP/TCP is almost instantaneous but in the event of, for example, a network disruption, the records being transferred will be lost.
    • The records being transfer are received by a job running on the JobScheduler providing the Background Service and written to the Background Service database.
  • The Background Service Automation batch mode is intended to be run at regular intervals by the JobScheduler providing the Background Service. This mode effectively provides a backup for the UDP/TCP transfer mode.
    • The Background Service Automation jobs read file transfer records that the YADE Client has saved in its persistent history file and writes these records to the Background Service database.

Background Service Configuration for the YADE Client

An initial configuration of the Background Service can be carried out using the XML Editor.

1) Schema Assignment

  • Open the XML Editor
  • The Assign schema form will suggest the last used schema. We recommend that you use the latest schema version that is available from YADE - Reference Documentation - XSD Schema Reference. Should you have downloaded this XSD schema to a local directory then the Open file button can be used to specify the required XSD schema.



  • Click on the Assign button to assign the schema.
  • The XML Editor will then open the minimum elements that have to be specified in the configuration as shown in the next screen shot.



  • The Configurations element xsi:noNamespaceSchemaLocation attribute entry field shown in the right hand pane of the XML Editor can be ignored for the purposes of this tutorial.
  • The configuration is saved in the XML format. 
    • The XML Editor can be configured to also generate a settings.ini file that can be used in YADE releases 1.10 and older. This file will be saved with the same basename as the XML file and the extension .ini in the same folder.
  • Additional elements can be added by right-clicking on visible elements.

2) Fragment Configuration

The next step is the configuration of fragments for the file transfer operation itself and for the Background Service. The parameters to be set can be seen in the screenshot below:

  • the ftp_server_2_local demonstration FTP fragment:
    • This fragment configuration has already been used in other tutorials in this series and downloads a number of files from an online FTP server.
  • the BackgroundServiceFragment Fragment:
    • This fragment specifies the JobScheduler service that the YADE Client is to send the file history information to immediately after completion of each file transfer operation.

3) Configuration of the Profile Elements

The file transfer profile is configured next.

  • Profile Configuration

    • The Profile element is almost identical to the simple ftp profile configuration used for in other tutorials in this series except that in this tutorial the target destination is a sub-folder of the jade_demo folder.
      The Profile configuration is shown in the screen shot below.

4) Configuration of the General History and Background Service Elements

The Transfer History and Background Service elements are configured next.

  • Note that:

    • Both these elements are located in the General branch of the Configuration. This branch functions in a similar way to a global include (and is, in fact, exported as a globals sub-profile when converted to the YADE settings.ini format). This allows the configuration to be specified for all profiles in a configuration.

    • Although more that one BackgroundServiceFragment can be specified in a Configuration, only a single BackgroundServiceFragmentRef element can be specified for use in the Configuration at any one time.

  • The Background Service Element

    • The Background Service fragment configured in the previous section is called using the BackgroundServiceFragmentRef element shown in the screenshot below:



  • The History Element

    • The generation of the persistent jade_history.csv file for the Background Service Automation is configured using the two parameters shown in the screenshot above: HistoryFile and HistoryFileAppendMode.
       

    • HistoryFile
      It is assumed that the YADE Client(s) and Background Service JobScheduler are installed on separate servers. For this situation the Background Service has a job chain which transfers files that individual YADE Clients have written on their local file system to the directory on the Background Service file system that is monitored by the jade_history_file_order job chain.
      • The Background Service monitors by default the directory:


    • HistoryFileAppendMode
      Setting this parameter to true means that a number of file transfer operations can be written to this file before it is processed by the Background Service. This has the advantage that it allows file transfer records to be 'collected' in the file and the reading of the file to be set to a time for less frequent file access.

Generation of a settings.ini File for Releases 1.10 and older

The XML Editor automatically generates a settings.ini file for use with releases 1.10 and older. This file has by default the same name and folder as the XML configuration file.

The settings.ini file generated for the configuration shown in the screenshots above is shown in the following code block.

YADE settings.ini file for the Background Service 
[globals]
history                             = C:\ProgramData\sos-berlin.com\jade\client_1.10\jade_history\jade_history.csv
HistoryFileAppendMode               = true

include                             = background_service_fragment@BS-local
[protocol_fragment_ftp@ftp_server_2_local]
protocol                            = ftp

host                                = test.sos-berlin.com
port                                = 21
user                                = demo
password                            = demo

[background_service_fragment@BS-local]
SendTransferHistory                 = true

BackgroundServiceHost               = localhost
BackgroundServicePort               = 4110
BackgroundServiceTransferMethod     = udp
BackgroundServiceJobChainName       = sos/jade/jade_history

[simple_ftp_sos_demo_bs]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_server_2_local
file_spec                           = .*
source_dir                          = /

target_protocol                     = local
target_dir                          = C:/Users/aa/jade_demo/c

Running a file transfer with Background Service 

It has already been mentioned above that the YADE Background Service parameters are effectively global settings that are called each time a file transfer operation is started from a configuration file where the Background Service is specified.

As shown in the The YADE Client Command Line Interface - Tutorial 1 - Getting Started article, the ftp_server_2_local demonstration operation is started by entering the following in the command line:

 

Windows
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="ftp_server_2_local"

The file transfer results

  • The file transfer can be followed in the command line window where the following information relevant to the Background Service will be shown. 

    Windows
    main INFO  20:06:43,576   (SOSFileList.java:689) ::writeTransferHistory 6 records written to history file 'C:\ProgramData\sos-berlin.com\jade\client_1.10\jade_history\jade_history.csv', HistoryFileAppendMode = true
main INFO  20:06:44,615   (SOSDataExchangeEngine.java:1455) ::sendTransferHistory 6 transfer history records sent to background service, scheduler = localhost:4110 ,job chain = sos/jade/jade_history, transfer method = udp
main INFO  20:06:44,646   (SOSDataExchangeEngine.java:890) ::printState SOSJADE_I_0101: Es wurden 6 Dateien ³bertragen
main INFO  20:06:44,655   (SOSDataExchangeEngine.java:391) ::showResult
  • In addition the jade_history.csv file in the ...jade\client_1.10\jade_history\ directory will have been updated with the six file transfer records for the ftp_server_2_local demonstration operation.

Background Service JobScheduler Configuration

The JobScheduler providing the Background Service has three job chains, each providing a separate part of the service:

  • jade_history
    • this job chain is started each time a file transfer record is sent by the YADE Client by either UDP or TCP
    • the job chain writes the file transfer record to the Background Service database
  • jade_history_receive
    • this job is used to collect history files from a number of servers that the YADE Client is operated for. This job makes use of SFTP to transfer the history files to the JobScheduler server. The history files are then processed by the jade_history_file_order.
    • this is an order job and a sample order can be found in the following JobScheduler file:
      • $SCHEDULER_DATA\config\live\sos\jade\jade_history_receive,receive.order.xml_sample
    • This order can be activated by by removing the _sample suffix from the file name.
    • The order can then be configured by opening the $SCHEDULER_DATA\config\live\sos\jade directory in JOE (the JobScheduler Object Editor), where it will appear against the ID/Title receive - receive as shown in the next screenshot.
  • jade_history_file_order
    • this job chain monitors a directory and by default is configured to process all history files landing in this directory.
      The default location of this directory is:
      • $SCHEDULER_DATA\jade_history\




    • Note that the order comes with a full set of sample parameters.
      • The screenshot shows the parameters after they have been modified to suit the Background Service configuration specified in this tutorial, where the history record file has been generated into a local directory and the transfer takes place from the source_dir shown in the screenshot to the ./jade_history target directory that is monitored by the jade_history_file_order job.
      • The source_user and source_password parameters have been deleted as they are not required for a local to local file transfer.
    • The sample order comes with a Run Time configuration for certain days of the month as can be seen in the screenshot.
    • This order will copy the history file generated by YADE to the target folder where it will be briefly visible until it is processed by the jade_history_file_order job chain before being deleted.
    • An instance of this order has to be started for each YADE Client in the Background Service.

History Viewer

The following screenshot shows the History Viewer output for the tutorial file transfer.

The screenshot shows the default file transfer operations list in the History Viewer along with the detailed information available. The Filter menu in the upper menu bar can be used to select lists each that have been completed. Further information about the use of the filter settings can be found in the YADE Transfer History article. All the parameters stored for each file are displayed in the right hand pane of the viewer when an individual file is selected as shown in the screenshot.

 

  • No labels