...

• File Watching is used to automatically start workflows in the event of a file arrival in a directory.
  • Agents watch directories for files and add a file order for each incoming file.
  • File orders are assigned workflows based on configurable rules.
• File Watching is a flexible means for enterprise application integration (EAI):
  • It enables e.g. 3rd-party applications to launch workflows simply by creating files.
  • It allows to trigger workflow execution on a number of Agents based on arrival of a file.

Feature Video

...

Trigger Files are typically zero-byte files that are created to trigger execution of a workflow. A workflow can remove such files at any point in time.

• Trigger files optionally can carry variables that are added to the file order in a workflow.
• For details see JS7 - JITL FileOrderVariablesJob.

Data Files are processed by jobs in a workflow e.g. , for example to import reporting data into a data warehouse. Such files are (re-) moved only after processing by their respective jobs.

...

The File Order Source is a configuration object that holds the directory and file name pattern to be watched for by an Agent and that assigns resulting file orders to a workflow.

• The Order ID of file orders for example is created like this: #<ISO date>#F<Seconds since Jan 1st 1970>-<ID of file order source>:<file name>, where:
  
  • #<ISO date># is the date of appearance of the file, e.g. 2021-03-17., enclosed by #. The date is calculated for the time zone assigned the File Order Source.
  • F is a qualifier to indicate a File Order Source.
  • <Seconds since Jan 1st 1970>-  is what it says and is followed by a hyphen.
  • <File Order source ID>: is the unique identifier of the File Order Source configuration object followed by a colon.
  • <file name> is the name of the incoming file.
• Such file orders can be considered triggers for workflow execution, which implies that the workflow can be executed by the monitoring Agent or by any other AgentsAgent.

File Order Sources are managed in the Configuration -> Inventory view like this:

Explanation:

• Workflow Name: A workflow is assigned to which an order is added per incoming file.
• Agent: An Agent is assigned that performs file watching. Standalone Agents and Agent Clusters can be assigned. In an Agent Cluster the active Director Agent performs file watching, Subagents do not perform file watching.
• Directory: The directory that the Agent watches for incoming files:
  • Unix: A path can be specified such as /tmp/incoming
  • Windows: A path can be specified with backslashes or forward slashes such as C:\tmp\incoming or C:/tmp/incoming
  • Unix, Windows: OS environment variables can be used that are known to the Agent, for example from its Instance Start Script. Environment variables and constant strings can be concatenated using the ++ operator and considering quoting for constant strings like this:
    • Unix: env('HOME') ++ '/incoming'
    • Windows: env('TMP') ++ '/incoming'
• Pattern: The pattern to match an incoming file file name pattern is not a wildcard expression such as *.csv, instead it represents a Java Regular Expression. The pattern has to match the path of an incoming file including the directory hierarchy, not just the file name. Consider the following examples:
  • match any files: .*
  • match files with a .csv extension: .*\.csv$a more elaborated expression to match ISO dates such as 2021-03-27 including support for long-short months and leap years: 
  • ^(?:(?=[02468][048]00|[13579][26]00|[0-9][0-9]0[48]|[0-9][0-9][2468][048]|[0-9][0-9][13579][26])match files that end with a date in yyyy-mm-dd format: .*\d{4}(?:(-|)(?:(?:00[1-9]|0[1-9][0-9]|[1-2][0-9][0-9]|3[0-5][0-9]|36[0-6])|(?:01|03|05|07|08|10|12)(?:\1(?:0[1-9]|[12][0-9]|3[01]))?|(?:04|06|09|11)(?:\1(?:0[1-9]|[12][0-9]|30))?|02(?:\1(?:0[1-9]|[12][0-9]))?|W(?:0[1-9]|[1-4][0-9]|5[0-3])(?:\1[1-7])?))?)$|^(?:(?![02468][048]00|[13579][26]00|[0-9][0-9]0[48]|[0-9][0-9][2468][048]|[0-9][0-9][13579][26])\d{4}(?:(-|)(?:(?:00[1-9]|0[1-9][0-9]|[1-2][0-9][0-9]|3[0-5][0-9]|36[0-5])|(?:01|03|05|07|08|10|12)(?:\2(?:0[1-9]|[12][0-9]|3[01]))?|(?:04|06|09|11)(?:\2(?:0[1-9]|[12][0-9]|30))?|(?:02)(?:\2(?:0[1-9]|1[0-9]|2[0-8]))?|W(?:0[1-9]|[1-4][0-9]|5[0-3])(?:\2[1-7])?))?)$-\d{2}-\d{2}$
• Delay: The delay in seconds for which a file is checked to be stable and does not change its size or timestamp. This guarantees that only files are picked up that have been completely written.

Using Files in Workflows

Using File Paths

For incoming files the JS7 File Order Source provides a built-in variable file that holds the path to the incoming file.

This variable is declared with the workflow like this:

Image Added

If the file variable is the only variable in a workflow then its declaration can be omitted. 

The variable can be used in subsequent jobs, for example to create an environment variable that is used from a job script:

• The below example assigns the environment variable FILE the value of the built-in file variable.
• This configuration step is the same for jobs executed with Unix or Windows.
  
  

Image Added

In the job script the environment variable FILE can be used like this:

• The below example shows a Windows job script using %FILE%.
• For Unix an environment variable is used from $FILE or ${FILE}.
  
  

Image Added

Using File Names

The built-in file variable holds the full path of the incoming file, for example /tmp/incoming/test.txt.

Users who want to use a file base name such as test.txt instead of the full path can configure their workflow like this:

Image Added

Explanation:

• Two workflow variables are declared:
  • The built-in file variable is declared without a default value.
  • A new filename variable is declared (an arbitrary name can be chosen for the variable).
    • The assigned value is a generic expression that extracts the file name from the incoming file's full path:
    • replaceAll( $file, '^.*[/\\]([^/\\]+)$', '$1' )
• The assignment of environment variables for a job can make use of both declared variables.
  
  

Image Added

In the job script the environment variables FILE and FILENAME can be used like this:

• The below example shows a Windows job script using %FILE% and %FILENAME%.
• For Unix environment variables are used from $FILE or ${FILE} and $FILENAME or ${FILENAME}.
  
  

Image Added

File Handling

Checking for Incoming Files

...

The best way for a client application to handle this situation is to make a file appear in an atomic transaction as e.g. , for example to use a temporary file name and to rename the file after it has been completely written. Agents trigger files by file name patterns and therefore will become aware of a file only when the file name path matches the pattern.

Should it not be possible for a client process that writes a file to rename the file after completion then the Agent implements a delay to check the steady state of an incoming file.

...

• The COMPLETED state does not indicate that previous jobs in the workflow were not successful. It can indicates that clean-up for the incoming file has failedstill is available having passed all nodes of the workflow.
• Users have to make completed file orders leave the workflow and move the remaining file to some other location.

...