Suspended orders are started after configuration changes and restart of JobScheduler

Observation

• A persistent Order is being suspended by a job error, at the same time the Order's configuration is manually being changed.
• JobScheduler does not change the suspended order's state, e.g. the Order remains suspended.
• If JobScheduler is restarted then the suspended Order is reset to the start node of the job chain and is started to the next point in time indicated by the run-time.
• Some users' might expect that a suspended Order remains suspended across a JobScheduler restart.

Maintainer Notes

• This feature works as designed. JobScheduler keeps track of the Order's state (running, suspended, stopped) using a persistent status flag in the database. When an Order is running JobScheduler also manages the Order's parameters and values in the memory.
• Handling of changes to configuration files
  • If the Order's configuration (order.xml) is being changed during a running/suspended state, JobScheduler does notice the change as replacement configuration. JobScheduler ensures that any replacement configuration will only take effect once the current execution of an Order is completed, e.g. the Order reaches to a job chain's end node.
  • Whenever JobScheduler is restarted with pending changes to an Order configuration, then the Order's temporary state is lost. During restart JobScheduler loads the configuration from disk (location ${SCHEDULER_DATA}/config/live).
  • During the restart JobScheduler checks for changed/updated configuration files, if configuration files on disk are newer than stored in the database then JobScheduler replaces the Order immediately.
  • Since any new Order configuration by default starts from a job chain's first node, any suspended and updated Order configuration will be reset to the start node.
• Explanation
  • The rule behind this behavior is that a newer file on disk should applied after restart which has more importance then the temporary state of an Order.
  • There are contradictory requirements
    • for a persistent Order JobScheduler should load the latest configuration during restart.
    • JobScheduler should restore the execution state of an order after an restart.
  • Suspending an order translates to "suspend the Order's execution" based on conditions, e.g. on error.
• Operation Handling
  • When there is an operational requirement to block an Order's execution for a longer duration, then the better approach will be to stop the entire job chain or a specific job chain node.