Double-check cluster node loss event

Current Behavior

• There can be situations when an Active Controller instance is unexpectedly slow in responding to the ClusterWatch (Proxy Service running in JOC Cockpit):
  • in case of an overloaded machine operating the Active Controller,
  • in case of software clock leaps (server time adjustments) of the machine operating JOC Cockpit that makes the Active Controller's responses appear from the past, See JS-2224.
  • in case of temporary network isolation of the Active Controller's machine. See JS-2092.
• The slow node (Active Controller instance) may emit a ClusterNodeLossEvent despite the peer (Standby Controller) does the same. In this situation the ClusterWatch may agree to fail-over as requested by the Standby Controller instance. As a result, both the originally active, but unressponsive Controller and the Standby Controller are active.
• Consequences of two active Controller nodes include potential double job execution with Agents and journal corruption.

Desired Behavior

• In case of an unresponsive Active Controller (missing or delayed responses to the ClusterWatch), the ClusterWatch will not immediately accept the Standby Controller's request to fail-over. Instead., the ClusterWatch will double-check after a delay if no response is received from the Active Controller (ClusterNodeLossEvent). Only then, the ClusterWatch will approve the Standby Controller's request to fail-over.

Delimitation

• Above scenario applies to established connections between JS7 products that are affected by slowness or temporary network isolation.
• The scenario does not apply to a situation when a connection is lost (Connection reset) or cannot be established (Connection refused).
• The scenario similarly applies to an Agent Cluster with the Active Controller holding the ClusterWatch role to orchestrate an Active Director Agent an a Standby Director Agent.