Make sure that file is completly written?

Problem:

file_order_source starts an order when the file is created, not when the file is ready.

In some file transfer scenarios the receiver of a file has no knowledge about when the sender creates a file. In case of a (very) large file, it is possible for the receiver to try to read a file before the sender has finished writing it. If the receiver then attempts to use the file at this moment, he will get a corrupted, incomplete file.

Solutions:

Using http://www.sos-berlin.com/doc/en/scheduler.doc/file_orders.xml file_order_source:

There are three ways you can use file_order_source:

• The sender creates a file named abc.txt~. When transmission is complete, the sender renames the file to abc.txt. You would use a regular expression such as ^.*\.txt$ to check for the presence of files.

• The sender creates a file named abc.txt. When it is ready, a second file with 0 byte will be created. The name of the second file is abc.txt.trigger. Here, you would use a regular expression such as ^.*\.txt.trigger$. Note that with this approach you have the disadvantage that the name of the trigger file is listed under scheduler_file_path, not the name of the file that should be executed.

• Set up a job chain where the file size is checked in the first node. Then carry out a setback if the file size is changing. This can be done with the job JobSchedulerExistsFile

See also: Directory Monitoring with File Orders

Using the JobSchedulerExistsFile job:

 This job has the advantage over {{file_order_source}} solutions that it allows the use of parameters, for example, for the name of the target directory and it allows you define your own polling rate.

The JobSchedulerExistsFile job also checks whether the file size is constant - i.e. Is the file still being written? - and will only proceed if the file size is not changing.

The JobSchedulerExistsFile job has three parameters to manage the check steady state behaviour.

• check_steady_state_of_files: If true, job will check the steady state.
• check_steady_state_interval: Interval in seconds between two checks
• steady_state_count: If set, this is maximum number of intervals. If the maximum is reached, the task will be ended with an error.

You can download example files covering both file_order_source and JobSchedulerExistsFile job solutions