Flexible Handling of late orders after Agent restart

Current Behavior

When JS7 components (Controller, Agents, and JOC Cockpit) are restarted after an outage or planned downtime, the system automatically processes all orders whose scheduled start times occurred during the downtime. These orders are treated as late orders and are immediately executed as soon as the services become available again.

While this behavior ensures that no scheduled task is missed, it can also lead to unintended operational consequences. In production environments, the immediate execution of multiple overdue orders may cause:

• Back‑to‑back or burst executions, creating sudden workload spikes
• Processing of outdated or irrelevant tasks, especially for time‑sensitive workflows
• Data inconsistencies, when delayed jobs operate on stale data
• Resource contention, as multiple late orders compete for system capacity

This default behavior may not always align with operational or business requirements.

Desired Behavior

This enhancement introduces a controlled way to handle late orders using new Controller-level settings. The goal is to make the system behave predictably after restarts or crashes, prevent late orders from running automatically, and give users clear visibility and manual control over what should happen.

A new configuration option, tentatively named ignore-late-orders, should be introduced at the Controller level. This setting determines whether late orders should run automatically after services restart. When set to false (the default), the system behaves as it does late orders are executed immediately. When set to true, late orders should no longer be executed automatically. Instead, all orders whose scheduled start time has passed should be moved from the “scheduled” state into the “pending” state when the Agent or Controller comes back online. Pending orders do not carry a scheduled start time and are clearly visible in the user interface under the Pending Orders view, where operators can review them and decide whether to start, cancel, or reschedule them. 

To avoid false classification of orders as late due to system time adjustments or clock synchronization issues, an optional tolerance parameter (for example, ignore-late-orders = 3s ) should be defined. This allows the system to determine precisely when an order should be considered late, accounting for small jumps in system time when some clock corrections occur.

In addition to handling normal restarts, the design should also account for Agent crash scenarios. For such cases, a separate configuration option, such as disable-after-failure, should be added. When enabled, this option ensures that no orders are executed automatically after the Agent recovers from a crash. This behavior is similar to manually disabling an Agent, but is triggered only when the system detects that the previous termination was abnormal. This protects the environment from executing any jobs under uncertain conditions after a crash.

When the Controller/Agent restarts:

• When the Controller or Agent restarts and ignore-late-orders = false (default):
     All late orders run automatically, maintaining current JS7 behavior.
• When the Controller or Agent restarts and ignore-late-orders = true:
     Late orders are moved to the pending state and do not run automatically.
     Operators manually decide whether to execute, cancel, or reschedule them.
• When disable-after-failure = true and the Agent has crashed previously:
  • The Agent will restart in a disabled state.
  • No orders will be executed automatically after the restart.
  • An operator must explicitly enable the Agent from JOC Cockpit, as the disabled status is recorded in both the Controller’s journal and the Agent’s journal.
  • This behavior ensures that the Agent does not resume processing until the operator intervenes.