Monitor service can throw a ConcurrentModificationException

Current Situation

• The Monitor Service is responsible for synchronization between the History Service and the Notification Service.
• One of its processing responsibilities is the evaluation and handling of Warnings for Jobs running longer than expected (longerThan checks).
• A ConcurrentModificationException can occur while processing longerThan entries:

  java.util.ConcurrentModificationException
      at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1855)
      at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
      at com.sos.joc.monitoring.model.HistoryMonitoringModel.handleLongerThan(HistoryMonitoringModel.java:...)
      ...

• One scenario is when the Notification Service stops (e.g. JOC Cockpit shutdown or Notification Service restart).
  • When the Notification Service stops, the Monitor Service persists the not yet processed longerThan entries to the database (so they are not lost and can be processed after restart).
  • If the Monitor Service is simultaneously busy processing longerThan entries, the parallel shutdown can lead to concurrent access to the same in-memory structure.
    • This may result in the above exception.
      • The processing of longerThan entries is sequential during normal execution, with the exception of the close/stop situation, which is invoked from a different thread.
      • If the Monitor Service is busy processing longerThan entries while the close operation is triggered, both threads may access and modify the same data structure concurrently.
• Impact:
  • Some longerThan notifications may be lost.
    • This is especially likely because the exception occurs during the evaluation phase, i.e. while execution time is being checked, before it is known whether the longerThan threshold is exceeded.

Desired Behavior

• The Monitor Service should handle such longerThan checks correctly and safely.

Test Instructions

• Create workflows with jobs using "Warn on longer execution" and configure a mechanism (e.g. sleep) so that execution time is exceeded.
• Configure Notification Service to generate notifications on warning.
• Run multiple job executions to create a high load scenario.
• Restart the Notification Service from the JOC Cockpit Dashboard during active processing.
• Check joc and service-monitor.log for errors or warnings.