JavaScript Job - File Size Monitor

Scope

• A job should monitor the size of a file, e.g. a log file.
• If the file size is not steady after some predefined timeout then an error should be raised or a predefined exit code or order state should be set.
• File size monitoring is implemented by a Monitor script that can be used by any job in a job chain.

Download, Installation and Operation

• Download
  • Download: file_size_monitor.zip
• Installation
  • Unzip the archive to the ./config/live folder of your JobScheduler Master.
  • A job chain job_chain1 is included with two jobs job1 and job2 and an order order1 with respective parameter settings.
  • The File Size Monitor is provided with the monitor_file_change monitor.

• Operation
  • Modify the parameter monitor_file_change_location of order1 to point to a file in your environment.
  • Start order1 from the JOC user interface.
    • Immediately modify and save the monitored file. Changes of the file size are checked every 10s for a max. duration of 30s.
    • After 30 seconds order1 will either be successfully moved to job2 or it will fail depending on the monitored file being considered steady within the given timeout.

Configuration

• The File Size Monitor can be configured from job or order parameters.

• The following parameters are used:

  Parameter	Required	Default	Description
  monitor_file_change_location	yes		The full path of a file that should be monitored.
  monitor_file_change_timeout	yes		The max. duration in seconds that repeated checks for steady file size are performed.
  monitor_file_change_interval	no	1	Specifies the number of seconds that the monitor sleeps between repeated checks of the file size.
  monitor_file_change_exit_code	no	1	Optionally sets the exit code of the current job to the specified value if the monitored files is not being found steady. This works for shell jobs exclusively.
  monitor_file_change_error_state	no		Optionally sets the order to the specified state if the monitored file is not being found steady

Implementation

Order Implementation

• An order can be used to carry the parameters for the File Size Monitor.

• The order can be implemented like this:

  Order Implementation Sample Expand source

  <order>
      <params >
          <param  name="monitor_file_change_location" value="/tmp/jobscheduler/file/some_changing_file.txt"/>
          <param  name="monitor_file_change_timeout" value="30"/>
          <param  name="monitor_file_change_error_state" value="error"/>
      </params>
  
      <run_time />
  </order>

Job Implementation

• A job that makes use of the File Size Monitor can reference the monitor by the <monitor.use> element.

• The job can be implemented as a shell job or as an API job for any of the supported languages, e.g.

  Job Implementation Sample Expand source

  <job  order="yes" stop_on_error="no">
      <script  language="shell">
          <![CDATA[
  echo hello world
          ]]>
      </script>
  
      <monitor.use  ordering="0" monitor="monitor_file_change"/>
  
      <run_time />
  </job>

   

  • Explanations
    • line 2 to 6: implements the job script
    • line 8: references the monitor script

Monitor Implementation

• The File Size Monitor is implemented as follows:

  Monitor Implementation Expand source

  function spooler_process_before() {
  
      var rc = true;
  
      // merge parameters from task and order
      var params = spooler_task.params;
      params.merge( spooler_task.order.params );
  
      // required parameters
      var monitorLocation = params.value( 'monitor_file_change_location' );
      var monitorTimeout = params.value( 'monitor_file_change_timeout' );
  
      // optional parameters
      var monitorInterval = params.value( 'monitor_file_change_interval' );
      var monitorExitCode = params.value( 'monitor_file_change_exit_code' );
      var monitorErrorState = params.value( 'monitor_file_change_error_state' );
   
      if ( !monitorLocation ) {
          spooler_log.error( '.. parameter missing: monitor_file_change_location' );
          return false;
      }
  
      if ( !monitorTimeout ) {
          spooler_log.error( '.. parameter missing: monitor_file_change_timeout' );
          return false;
      } else {
          monitorTimeout = parseInt( monitorTimeout );
      }
  
      if ( !monitorInterval ) {
          monitorInterval = 1;
      } else {
          monitorInterval = parseInt( monitorInterval );
      }
  
      if ( !monitorExitCode ) {
          monitorExitCode = 1;
      } else {
          monitorExitCode = parseInt( monitorExitCode );
      }
  
      // let's use a Java package for file operations
      var monitorFile = new java.io.File( monitorLocation );
      if ( !monitorFile.isFile() ) {
          spooler_log.error( 'file does not exist: ' + monitorLocation );
          return false;
      } else {
          spooler_log.info( '.. monitoring file for max. ' + monitorTimeout + ' seconds: ' + monitorLocation );
      }
  
      var currentDuration = 0;
      var fileSize = 0;
      var lastFileSize = 0;
  
      do {
          lastFileSize = monitorFile.length();
          spooler_log.info( '.. current file size: ' + lastFileSize + ' bytes' );
          java.lang.Thread.sleep( monitorInterval * 1000 );
          fileSize = monitorFile.length();
          currentDuration = currentDuration + monitorInterval;
      } while ( (fileSize != lastFileSize) && (currentDuration < monitorTimeout) )
  
      if ( fileSize === lastFileSize ) {
          spooler_log.info( '.. file size is steady after ' + currentDuration + ' seconds (' + fileSize + ' bytes): ' + monitorLocation );
      } else {
          spooler_log.info( '.. giving up as file size is not steady after ' + currentDuration + ' seconds (' + fileSize + ' bytes): ' + monitorLocation );
          rc = false;
  
          if ( monitorExitCode ) {
              spooler_log.info( '.. setting task exit code: ' + monitorExitCode );
              spooler_task.exit_code = monitorExitCode;
          }
  
          if ( monitorErrorState ) {
              spooler_log.info( '.. moving order to error state: ' + monitorErrorState );
              spooler_task.order.state = monitorErrorState;
          }
      }
  
      return rc;
  }

   

  • Explanations
    • line 1: the monitor implements the spooler_process_before() function, i.e. it is executed for orders (not for standalone jobs) before the order is processed by the associated job.
    • line 6-7: the monitor parameters are merged from task and order parameters. Precedence is granted to order parameters.
    • line 10-16: parameter values are assigned to local variables.
    • line 18-28: the required parameters are checked and an error is raised if parameters are missing.
    • line 30-40: default values for optional parameters are assigned.
    • line 43-49: the monitor makes use of the Java File class as JavaScript does not provide file operations. Integrating Java with JavaScript works seemlessly as both are provided by the JVM. This integrations allows to use any Java classes from JavaScript. The monitor checks the existence of the monitored file and will raise an error if the file does not exist.
    • line 51-61: a loop is implemented that considers changes in the file size for the max. duration of repeated checks. Between checks a sleep interval is implemented.
    • line 63-78: finally if the file size is found to be steady the respective informative message is written to the log and otherwise the return code is set accordingly.