Controller crash with NullPointerException during cluster switchover/failover under high volume of scheduled/waiting orders

Current Situation

During a JS7 Controller cluster switchover/failover, the newly active Controller restores and reschedules all existing orders.

If a large number of orders are in Waiting or Scheduled state, or are due to be triggered around the same time, the Controller attempts to process them simultaneously. This creates a high level of concurrent processing during startup.

Under these conditions, the Controller may fail to start correctly. In observed cases, the process terminates with a NullPointerException:

• ERROR js7.journal.JournalingActor - »ControllerOrderKeeper« java.lang.NullPointerException: Cannot invoke "cats.effect.unsafe.TimerHeap$Node.triggerTime()" because "y" is null

During switchover, the Primary Controller transitions to standby while the Secondary Controller becomes active. When the JOC Cockpit displays Cluster Status: Switched Over, the Secondary Controller crashes with the above exception, leaving the cluster without an active Controller.

Please find the attached logs for your reference.

Desired Behavior

During JS7 Controller cluster switchover/failover, the newly active Controller must reliably restore all existing orders (including Waiting and Scheduled states) under high concurrency without runtime exceptions. The Controller must not crash and should complete startup successfully.

Workaround

Restart the JOC, Controller, and Agent services.

Maintainer Notes

• The issue is considered resolved as no NPE occurs and as the Controller restarts in case of a Scala TimerHeap Exception.
• At the same time, the root cause is not in scope of the issue as it is based on resource bottlenecks:
  • If the Controller is assigned a single, not too powerfull CPU, for example, in a containerized environment, then the TimerHeap Exception will occur.
  • The same can happen using more CPUs that do not provide sufficient power to manage a situation of heavy load such as switch-over with thousands of orders. It remains the fact that resource consumption depends on the number of orders and days for which the Daily Plan is created. Technically, there is little reason to plan ahead the Daily Plan for more than 7 days as Daily Plan Projections offer a better means to check future execution dates for months or years ahead.