Improve resilience of Controller/Director Agent Cluster in case of hardware clock leaps

Current Situation

• The Controller Cluster and Director Agent Cluster rely on synchronization of server clocks, find details from the Wiki page.
• In case of clock leaps, certain thresholds exist for the cluster's behavior:
  • 3s: cluster will catch up
  • 10s: cluster is affected, but usually will recover
  • 20s cluster will fail

Problem
If the threshold value for clock leaps is exceeded and the hardware clock of the Active Controller instance is slower than that of the Standby Controller instance, then the Cluster Watch (JOC Cockpit) and Standby Controller will initiate fail-over as they consider the messages of the Active Controller being outdated. 

However, the Active Controller is still alive (while all others consider it dead due to the time difference) and is still connected to Agents.

At the same time, the Standby Controller becomes active and starts exchanging events with the Agents. This state does not last long, after 1-3s the Cluster Watch will instruct the Active Controller (if reachable) to become standby. But: in the mean-time, the (former) Active Controller possibly has exchanged events with Agents that the new Active Controller does not know (and vice versa). This can result in journal corruption which is indicated by warnings such as “inapplicable event”.

The cluster no longer couples, if both Controller instances receive events from Agents for the moment they are active at the same time. Agent responses arrive to requests made by the other Controller, there is nothing a Controller instance can do about a response for which it didn't send the request. Both Controller instances assume they are on standby as they do not receive current events and neither instance will take the lead in the Cluster. This means the Cluster is inoperable.

Desired Behavior

• Resilience
  • When the standby instance intends to become active as there are no responses from the active instance in good time, then the Cluster Watch (active JOC Cockpit for Controller Cluster or active Controller instance for Director Agent Cluster) will ask the currently active instance before consenting to fail-over.
    • A timeout of 20s is applied. If the timeout is exceeded, then fail-over occurs.
    • Any errors from connections to the active instance result in fail-over.
• This applies to both Controller Cluster and Agent Director Cluster.