Raise warnings in Controller/Director Agent Cluster in case of hardware clock leaps

Current Situation

• The Controller Cluster and Director Agent Cluster rely on synchronization of server clocks, find details from the related article.
• In case of clock leaps, certain thresholds exist for the cluster's behavior:
  • 3s: cluster will catch up
  • 10s: cluster is affected, but usually will recover
  • 20s cluster will fail

Problem
If the threshold value for clock leaps is exceeded and the hardware clock of the Active Controller instance is slower than that of the Standby Controller instance, then the Cluster Watch (JOC Cockpit) and Standby Controller will initiate fail-over as they consider the messages of the Active Controller being outdated. 

The similar problem applies for a Director Agent Cluster with the active Controller instance acting as Cluster Watch.

Desired Behavior

• Logging
  • Clock leaps will be logged . If thresholds are exceeded then warnings and errors will be stated with the log.
• This applies to both Controller Cluster and Agent Director Cluster.