Limit job streams thread consumption

Current Situation

• Job Streams use Java futures and threads to manage events and job starts.
• In a number of situations job streams will not terminate running threads:
  • Technical Implications
    • Job Streams are running as a plugin in the Master. Thread consumption of Job Streams counts towards the JVM used by the Master. If the number of threads increases then the Master's internal web server can become slow to respond to requests of JOC Cockpit or of the Job Streams plugin or might not respond at all if no new threads can be created. This results in "read timeout" error messages that are visible from the Master's scheduler.log file.
    • Threads count towards the heap space of the JVM. If the heap space is filled up then the Java Garbage Collector will try even more aggressively to free the little space left in the heap. This results in high CPU consumption.
    • If the Java heap space is filled up and no new threads can be created then the JVM crashes.
  • User Observations
    • Spontaneous job execution: Threads unblock at an undesired point in time and cause job starts.
    • Thread starvation: we find up to 30000 threads filling up the Java heap space and crashing the JVM

Desired Behavior

• The Job Streams implementation carefully cleans up any Java futures and threads. The number of such objects is limited to a constant value.

Workaround

• The immediate workaround is to restart the Master more frequently. Particularly in case of deployment of a larger number of jobs used by job streams a restart is recommended.

Patch

A patch is available from https://download.sos-berlin.com/patches/patch_20221104_JS-2027-for1.13.16.jar

• Download the jar file
• copy the file to $SCHEDULER_HOME/lib/patches
• restart JobScheduler Master