You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Problem

Users might observe a situation when orders seem to have travelled through time - carrying a more recent start time than end time in the JS7 - Order History:

Analysis

A possible reason for non-matching timestamps is the fact that the servers involved in job scheduling do not use a the same system time.

  • Typically in Unix and Windows environments a time service such as NTP is used to synchronize a server's clock from an external time server.
    • Should the time service not be operational for some time then a server's clock will become more and more inaccurate.
    • Small delays of milliseconds can add up to a number of seconds and more after some time.
  • The problem starts from a situation when servers used for the Controller and Agents no longer have their clock synchronized:
    • When an order is added then the Controller knows the start time that an order is scheduled for. However, it's the Agent that reports completion of the last job in a workflow as the order's end time. JS7 - Resilience requires that an Agent can act autonomously and can report back the end time of an order. In a situation when the Agent cannot be contacted by its Controller, for example in case of network outages, the Agent will report back to the Controller at a later point in time. 
    • If the Controller's clock is behind the Agent's clock then an order that requires less time than the delay between both servers' clocks
      • will show with the Order History a start time that is behind the end time,
      • will - even more confusingly - show non-matching timestamps in the order log.
    • Even in cases of smaller delays the results displayed in the Order History will not be accurate.
  • This does not apply to start times and end times of respective jobs that are reported back by the same Agent.

Resolution

  • It is essential that the servers used for JOC Cockpit and Controller use synchronized time.
    • In case of a JS7 - Controller Cluster the servers involved have to use synchronized clocks.
    • The same applies to the server operating the Cluster Watch Agent.
  • It is recommended to use synchronized clocks for servers that are operating Agents.
    • However, except for the confusion from logs and display of the history there is no detrimental effect in case of non-matching system times.



  • No labels