Notification Queue Is Not Persisted and Causes Notification Loss After JOC Restart

Current Situation

• Notifications are processed by a single-threaded queue and executed sequentially. When the notification generation rate exceeds the processing rate, the queue grows continuously, as notifications are added faster than they can be processed.
• The notification queue is maintained only in memory and is not persisted during JOC shutdown. Consequently, any notifications still waiting in the queue at the time of shutdown are lost and are not recovered after JOC restart.

Desired Behavior

• Pending notifications should be retained across JOC restarts. Notifications waiting in the queue during shutdown should be persisted and restored after startup, allowing processing to continue without loss.
• The notification processing rate should be able to keep pace with the generation rate so that the queue does not grow unbounded, for example through parallel processing.

Maintainer Note

• CONFIGURATION -> Notifications
  • Two new settings/attributes have been introduced on the Notifications element:
    • parallelism (default: 10)
      • Specifies the maximum number of notification monitor executions that may run in parallel.
      • Increasing this value can improve notification throughput in high-load situations by allowing multiple monitor actions (for example email or command monitors) to execute concurrently instead of waiting for previous executions to finish.
      • A higher value can reduce notification delays, especially when monitor actions are slow or block while waiting for external systems.
      • However, excessive parallelism may increase CPU, memory, database, network and operating system resource usage and can affect the overall responsiveness of the application.
    • retry_incomplete_on_startup (default: false)
      • Specifies whether notification monitor executions that were interrupted during application shutdown should be retried on the next startup.
      • This can be useful if notification processing was still active when the application stopped and some monitor executions did not complete.
        • This can happen, for example, in high-load situations when a monitor command script requires several seconds to complete.
          • Notification processing is performed in parallel, however notification completion depends on the execution time of the monitor action.
          • During shutdown the monitor service waits up to 60 seconds for active notification executions to finish.
            • Notifications that have not yet started processing are persisted and restored on the next startup.
            • Notifications that are actively executing can be interrupted and may end with an InterruptedException.
      • Risks of enabling retry_incomplete_on_startup.
        • An InterruptedException does not reliably indicate the actual execution state of the monitor action.
          • The interruption may occur at any stage of processing and it is generally unknown how much of the intended work has already been completed.
        • Retrying such notifications can result in duplicate actions.

Test Instructions

• See TEST-JOC-2242.zip