Skip to end of metadata
Go to start of metadata

Problem

Users might observe a situation where orders seem to have travelled through time - showing a more recent start time than end time in the JS7 - Order History:

Analysis

A possible reason for non-matching timestamps could lie in the fact that the servers involved in job scheduling do not use a the same system time.

  • Typically in Unix and Windows environments a time service such as NTP is used to synchronize a server's clock from an external time server.
    • If the time service is not operational for some time then a server's clock will become more and more inaccurate.
    • Small delays of milliseconds can add up to a number of seconds and more after some time.
  • The problem starts with a situation where servers used for the Controller and Agents no longer have their clocks synchronized:
    • When an order is added then the Controller knows the start time that an order is scheduled for. However, it's the Agent that reports completion of the last job in a workflow as the order's end time. JS7 - Resilience requires that an Agent can act autonomously and can report back the end time of an order. In a situation when the Agent cannot be contacted by its Controller, for example in case of network outages, the Agent will report back to the Controller at a later point in time. 
    • If the Controller's clock is behind the Agent's clock then an order that requires less time than the delay between both servers' clocks:
      • will show with the Order History a start time that is behind the end time,
      • will - even more confusingly - show non-matching timestamps in the order log.
    • Even in cases of smaller delays the results displayed in the Order History will not be accurate.
  • This does not apply to start times and end times of jobs which are reported back by the same Agent.

Resolution

  • It is essential that the servers used for JOC Cockpit and Controller use synchronized time.
    • In case of a JS7 - Controller Cluster the servers involved have to use synchronized clocks.
    • The same applies for the server operating the Cluster Watch Agent.
  • It is recommended that synchronized clocks are used for servers which are operating Agents.
    • However, except for the confusion from logs and display of the history there will be no detrimental effect in the event of non-matching system times.



  • No labels