Introduction

• File Watching is used to automatically start workflows in the event of a file arrival in a directory.
  • Agents watch directories for files and add a file order for each incoming file.
  • File orders are assigned workflows based on configurable rules.
• File Watching is a flexible means for enterprise application integration (EAI):
  • It enables e.g. 3rd party applications to launch workflows simply by creating files.
  • It allows to trigger workflow execution on a number of Agents based arrival of a file.

Feature Video

This video explains file watching with JS7.

Trigger Files and Data Files

Trigger Files are typically zero-byte files that are created to trigger execution of a workflow. A workflow can remove such files at any point in time.

Data Files are processed by jobs in a workflow e.g. to import reporting data into a data warehouse. Such files are (re-) moved only after processing by their respective jobs.

For both Trigger Files and Data Files the workflow is responsible for removing an incoming file before completion of the workflow.

File Order Sources for Workflows

The File Order Source is a configuration object that holds the directory and file name pattern to be watched for by an Agent and that assigns resulting file orders to a workflow.

• The Order ID of file orders for example is created like this: #<ISO date>#F<Seconds since Jan 1st 1970>-<ID of file order source>:<file name>, where:
  
  • #<ISO date># is the date of appearance of the file, e.g. 2021-03-17., enclosed by #. The date is calculated for the time zone assigned the File Order Source.
  • F is a qualifier to indicate a File Order Source.
  • <Seconds since Jan 1st 1970>-  is what it says and is followed by a hyphen.
  • <File Order source ID>: is the unique identifier of the File Order Source configuration object followed by a colon.
  • <file name> is the name of the incoming file.
• Such file orders can be considered triggers for workflow execution, which implies that the workflow can be executed by the monitoring Agent or other Agents.

File Order Sources are managed in the Configuration -> Inventory view like this:

Explanation:

• The file name pattern is not a wildcard expression such as *.csv, instead it represents a Java Regular Expression. Consider the following examples:
  • match any files: .*
  • match files with a .csv extension: .*\.csv$
  • a more elaborated expression to match ISO dates such as 2021-03-27 including support for long-short months and leap years: ^(?:(?=[02468][048]00|[13579][26]00|[0-9][0-9]0[48]|[0-9][0-9][2468][048]|[0-9][0-9][13579][26])\d{4}(?:(-|)(?:(?:00[1-9]|0[1-9][0-9]|[1-2][0-9][0-9]|3[0-5][0-9]|36[0-6])|(?:01|03|05|07|08|10|12)(?:\1(?:0[1-9]|[12][0-9]|3[01]))?|(?:04|06|09|11)(?:\1(?:0[1-9]|[12][0-9]|30))?|02(?:\1(?:0[1-9]|[12][0-9]))?|W(?:0[1-9]|[1-4][0-9]|5[0-3])(?:\1[1-7])?))?)$|^(?:(?![02468][048]00|[13579][26]00|[0-9][0-9]0[48]|[0-9][0-9][2468][048]|[0-9][0-9][13579][26])\d{4}(?:(-|)(?:(?:00[1-9]|0[1-9][0-9]|[1-2][0-9][0-9]|3[0-5][0-9]|36[0-5])|(?:01|03|05|07|08|10|12)(?:\2(?:0[1-9]|[12][0-9]|3[01]))?|(?:04|06|09|11)(?:\2(?:0[1-9]|[12][0-9]|30))?|(?:02)(?:\2(?:0[1-9]|1[0-9]|2[0-8]))?|W(?:0[1-9]|[1-4][0-9]|5[0-3])(?:\2[1-7])?))?)$

File Handling

Checking for Incoming Files

Timing

The JS7 Agent applies the following procedure for the timely acceptance of incoming files:

• For Linux environments the Inotify interface is used. This notifies the JS7 Agent in near real-time about incoming files.
• For other Unix environments including MacOS, AIX etc. the Agent performs polling every 2s based on the implementation provided with the relevant Java Virtual Machine.
• For Windows environments the relevant API is used, and notifies the JS7 Agent in near real-time.

Steady State Check

When an incoming file arrives in a directory then the file might not have been completely written at the point in time of appearance.

• In Windows environments, files that are being written by a process cannot be accessed by jobs.
• In Unix environments, parallel read and write operations to files by jobs and processes are possible - but not desired, as results are unpredictable.

The best way for a client application to handle this situation is to make a file appear in an atomic transaction as e.g. to use a temporary file name and to rename the file after it has been completely written. Agents trigger files by file name patterns and therefore will become aware of a file only when the file name matches the pattern.

Should it not be possible for a client process that writes a file to rename the file after completion then the Agent implements a delay to check the steady state of an incoming file.

• An interval (default: 2 seconds) can be specified. The Agent will wait for this time and then check if the time stamp or the size of the file changed.
• If the file has changed then the Agent will wait for the next interval to before applying the same check.

Ghost Appearance

If a file appears then the Agent will create a file order. If the file disappears later on then this has no impact on the file order. However, if a file with the same name appears once again while the current file order is in progress then the Agent will prepare a new file order that will be visible only after the initial file order has completed the workflow.

It is not considered good practice for a client application to make files with the same name appear and disappear in short sequence. However, the situation will be handled by the Agent that creates additional file orders.

Behavior with Outages

If an Agent was not available when a file has been added to a directory then after start of the Agent it will pick up the file and will create a file order.

If the Controller is not available at the point in time when a file order is created then the Agent:

• can process a workflow for the file order provided that all jobs in this workflow are assigned to this Agent,
• has to wait for the Controller to become available in order to forward the file order.

Moving Files and Removing Files on Workflow Completion

JS7 does not move or remove files, it's the workflow's responsibility to guarantee that on completion of a workflow the incoming file is no longer present.

• A job in a workflow can:
  • move an incoming file to an archive location that is not subject to file watching,
  • remove an incoming file.

If an incoming file is still present on completion of the workflow then the file order is moved to a COMPLETED state.

• The COMPLETED state does not indicate that previous jobs in the workflow were not successful. It indicates that clean-up for the incoming file has failed.
• Users have to make completed file orders leave the workflow and move the remaining file to some other location.