Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 34 Next »

 

Introduction

The selection & handling of the objects being transferred can be divided into a number of areas:

  • File selection - which broadly covers the specification of features that are related to the source side of the file transfer operation such as filtering, polling and directives.
  • File handling - which broadly covers features that are implemented on the target side of the operation such as atomic transfer and the compression, checking and appending of files.
  • Renaming - which is also related to the source side of the file transfer operation and - in contrast with the file selection and file handling features - is protocol-specific.

Checking files for completeness is handled in the next page.???

Configuring File Selection, Handling and Renaming USE AS INCLUDE ON ALL PAGES?

We recommend that the XML Editor is used to generate the JADE configuration file - it uses the JADE XSD schema to suggest valid entries during runtime and thereby helps to considerably reduce errors.

Information about how to call a profile from the command line can be found in:

The XSD schema is hierarchical and the relevant parts of this hierarchy are shown in the sections below. This information can then be used in the XML Editor to navigate to the appropriate form required to enter configuration parameters.

File Selection

How to specify file selection

File selection is specified in the Profiles branch of the configuration and can be applied to for all the Operations that can be defined for JADE: Copy, Move, etc.

File selection is a source side operation, and the JADE file transfer options are all specified as child elements of the SourceFileOptions element.

This configuration hierarchy is shown for the Copy operation as follows:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
          • SourceFileOptions
            • Selection (required)
            • optional file selection parameters
        • CopyTarget
          • copy target parameters
      • Move
      • etc.

Note that all the optional file selection parameters apply to all files matching the selection criteria.

File Selection Functions

JADE comes with a comprehensive set of file selection functions:

  • Select an individual file or folder: 

    • FilePathSelection
      • This parameter is used to specify an individual file or a folder from which all files will be processed.

  • Select a dynamic number of files from a folder:

    • FileSpecSelection
      • Selects all files from a folder that match a regular expression
      • Can be used to select files recursively

  • Select files from a list:

    • FileListSelection
      • This parameter is used to specify a number of files for processing.

These functions are specified using one of the child elements of the Selection element.

The element hierarchy for the Selection element when a Copy operation is specified is:

  • Profile 
    • Operation 
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions (required)
              • Selection (required)
                • FilePathSelection
                  • FilePath (required)
                  • Directory
                • FileSpecSelection
                  • FileSpec (required)
                  • Directory
                  • Recursive
                • FileListSelection
                  • FileList (required)
                  • Directory

Exact definitions for each file selection element and the interaction between elements can be found in the Selection Parameter Reference.

Checking files for completeness (CheckSteadyState)

The CheckSteadyState option can be used as a method for checking that a file is fully available at the file transfer source.

The file size is measured at regular intervals and until a steady state is determined.

Note that this method of determining whether a file is fully available is not 100% reliable. This is discussed in detail in the CheckSteadyState parameter reference alongside alternative approaches for avoiding corrupt files and a description of the three child elements of the CheckSteadyState parameter: CheckSteadyStateInterval, CheckSteadyStateCount and CheckSteadyStateErrorState.

The CheckSteadyState parameter is a child element of SourceFileOptions as shown: 

  • Profile  
    • Operation  
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions (required)
              • Selection (required)
              • CheckSteadyState (option)
                • CheckSteadyStateInterval
                • CheckSteadyStateCount
                • CheckSteadyStateErrorState

Behavior when transfer criteria are not met (Directives) ÜBERARBEITEN

Directives can be seen as optional parameters that specify how JADE is to generally react at the file transfer source.

The Directives element itself is a child of the SourceFileOptions elements:

  • SourceFileOptions
    • Selection (required)
    • Directives (optional)

      • DisableErrorOnNoFilesFound

      • TransferZeroByteFiles

    • Other optional source file elements

The directives are:

  • DisableErrorOnNoFilesFound

    • The DisableErrorOnNoFilesFound parameter allows the default behavior of JADE to be changed.
      Note that the use of this parameter can modify the behavior of JADE when TransferZeroByteFiles is set.

  • TransferZeroByteFiles

    • The TransferZeroByteFiles parameter specifies how JADE is to handle zero byte files - i.e. whether they are to be transferred and processed by subsequent commands.
      See the Parameter Reference for a details about how this parameter interacts with the Remove and DisableErrorOnNoFilesFound parameters.

Polling for Files

JADE can use Polling to wait for incoming files. The files to be polled are specified with the Selection element and polling can be is used to specify, for example, how often and how long should be polled.

Polling only continues until the polling criteria have been met (i.e. one or a minimum number of files have been found).

File transfer is started once polling has stopped.

When JADE is run as a job in the JobScheduler a job node can be specified for execution in case polling is unsuccessful.

A detailed list of the polling parameters that can be set and their definitions can be found in the Polling section of the Parameter Reference.

The Polling parameter is a child element of SourceFileOptions as shown: 

  • Profile  
    • Operation  
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions (required)
              • Selection (required)
              • CheckSteadyState (option)
              • Polling (option)
                • PollInterval
                • PollTimeout
                • etc
              • Other optional source file elements

Restricting the Files to be transferred (multiple parameters)

The following file filtering options can be used to filter the files that are to be transferred:

  • FileAge

    • FileAge sets the minimum and/or maximum file age

  • FileSize

    • FileSize sets the minimum and/or maximum file size

  • ResultSet

    • The ResultsSet parameter handles the result set of a Selection of files from the source host. The result set can be written into a file or, when JADE is run as a JobScheduler job, checked against an expectation.

  • SkipFiles

    • SkipFiles allows the first and/or last files found to be omitted from transfer.

  • MaxFiles

    • The MaxFiles parameter allows the maximum number of files to be transferred to be set. All other files will be ignored.
    • Note that it is not possible to specify which files will be ignored should the value set for this parameter be exceeded. It therefore only makes sense to use this parameter in particular situations such as when the contents of a directory will be repeatedly polled. INFO TO PARAM REF

A detailed description of the file filtering elements listed above can be found in the SourceFileOptions Parameter Reference.

This configuration hierarchy relevant for the SourceFileOptions parameters where a Copy operation is specified is:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
            • SourceFileOptions
              • Selection (required)
              • FileAge
              • FileSize
              • ResultSet
              • SkipFiles
              • MaxFiles
              • Other optional file selection elements

File Handling

File handling is used here to refer to operations that are carried out target side of a transfer operation.

How to specify file handling

File handling is specified in the Profiles branch of the configuration and can be applied to those Operation elements that can be defined for JADE: Copy, Move, etc.

File selection is a target side operation, and the JADE file transfer options are all specified as child elements of the TargetFileOptions element.

This configuration hierarchy is shown for the Copy operation as follows:

  • Profile
    • Operation
      • Copy
        • CopySource
          • CopySourceFragmentRef
          • SourceFileOptions
            • Selection (required)
            • optional file selection parameters
        • CopyTarget
          • CopyTargetFragmentRef (required)
          • Directory (required)
          • TargetFileOptions (option)
            • AppendFiles (option)
            • etc.
      • Move
      • etc.

Note that all the optional file selection parameters apply to all files matching the selection criteria.

The file handling options

Appending one file to another

The AppendFiles element can be used to specify whether files are to be appended to other files (i.e. added to the end of a file already existing in the target directory)

AppendFiles is specified as a child element of the TargetFileOptions element, which itself can be a child element of the CopyTarget or MoveTarget elements as shown schematically in the JADE User Manual - File Selection, Handling and Renaming section above:

Files are appended to files with the same name. VERIFY!

Note that if the AppendFiles element parameter is set to true then DisableOverwriteFiles (another optional child of TargetFileOptions) will be ignored.

Compress files

JADE can compress individual files before writing them to the target directory. VERIFY!

The CompressFiles element specifies whether files should be compressed with a zip algorithm before saving to the target system.

Note: JADE is not able to combine files at the source to a single compressed file before transfer (see the relevant change Issue for details).

Could this be done using SFTP and preprocessing?VERIFY!

Configuring error handling

The default configuration for JADE is that an error will be logged if a transfer should take place but no files are found

This behavior can altered with the DisableErrorOnNoFilesFound element, which is a child of the Directives element, which itself is a child of the SourceFileOptions elements.

These elements are reusable - that is they can be specified for a number of source parent elements such as CopySource or GetListSource.

See the DisableErrorOnNoFilesFound parameter reference article for more detailed information about this element, in particular on its interaction with the FileSpec, FilePath and FileList elements.

File Renaming

File renaming is specified using the Rename element, and can be applied at either the source or target sides of a file transfer.

Renaming can be applied to operations that are carried out using all protocols JADE can currently use (Release 1.11.x). 

File renaming is protocol-specific and is specified as a child element of FragmentRef elements such as FTPFragkentRef, SMBFragmentRef and LocalSource and LocalTarget elements in Profiles.

The element hierarchy related to a Rename parameter specified for a copy to a local target operation is:

  • Copy
    • CopySource
      • child elements ...
    • CopyTarget
      • CopyTargetFragmentRef
        • LocalTarget
          • Rename (optional)
            • ReplaceWhat
            • ReplaceWith

The Rename element requires that two child elements are defined:

  • ReplaceWhat - a regular expression that allows selection of parts of file names.
  • ReplaceWith - complex replacement patterns using substitution masks and capturing groups can be used.

A detailed description of the possible file name replacement patterns can be found in the Rename element parameter reference.

See also Atomicity.

 

 

 

  • No labels