Use of Virtual Threads with Java 21 blocks processing of jobs

Current Situation

• With release 2.7.0 support for Virtual Threads was added which is available when using Java 21.
• With release 2.7.2 feature JS-2148 was introduced that holds PIDs of job processes in a Crash PID File to allow a watchdog process to terminate job processes in case of crash of the Agent.
• The following problems can occur:
  • A race condition can occur if a number of job processes are starting in parallel and the Agent tries to store/remove PIDs to/from the Crash PID File.
    • The problem becomes evident from log output like this that indicates a delay when accessing the file:

      2024-12-04T15:34:50,996 info  js7.base.utils.AsyncLock - †53 js7.launcher.crashpidfile.IndexedRecordSetImpl#lock: js7.launcher.crashpidfile.IndexedRecordSetImpl#manipulate is still waiting for 40s (3 queued)...
      

• • Job scripts are written to disk with a delay of 5s.
    • This happens when a workflow is deployed and is executed for the first time after deployment.
    • This happens when the Agent is restarted.
  • Jobs start with a delay of 2s.
• The problems depend on resource usage of Virtual Threads and might not be reproducible for all users and not for all scheduling situations.

Desired Behavior
Virtual Threads will not be used by JS7 until a more complete implementation is available from later Java releases.

Workaround
Users who face the above problems should fall back to use of Java 17 when using releases 2.7.0, 2.7.1, 2.7.2.

Users who cannot immediately switch back to Java 17 can apply the following Java option on start-up of the Agent:

-Djdk.virtualThreadScheduler.maxPoolSize=1000000