History Service should manage larger log output

Current Situation

• Jobs can write log output of arbitrary size to the job's stdout/stderr channel. The JS7 Agent and Controller handle log output based on a streaming implementation that can handle this size.
• The JOC Cockpit is affected by larger log output as its History Service has to write log files to disk, to compress log files and to add them to the database. Compressing zip files comes at a cost for memory and can raise OutOfMemoryError exceptions that make JOC Cockpit unavailable.

Desired Behavior

• The JOC Cockpit History Service works with two threshold settings:
  • Setting for applicable size of a task log (name: log_applicable_size, default: 500 MB). A task log that exceeds this size will be handled like this:
    • The first 100 KB and the last 100 KB of the log file will be used as the task log.
      • A banner information is added to the begin of log output:
        • LOG OUTPUT EXCEEDS APPLICABLE SIZE OF <log_applicable_size> MB AND IS TRUNCATED TO THE FIRST AND LAST 100 KB.
      • The truncated task log will be compressed and added to the database.
    • The original log file will be removed.
    • Occurrence of log output exceeding this threshold raises a warning, however, the task execution history and truncated log output are available from the JOC Cockpit API and GUI.
  • Setting for maximum size of a task log (name: log_maximum_size, default: 1000 MB). A task log exceeding this size will be handled like this:
    • The History Service will stop to write larger log files. Log output exceeding the threshold value is ignored.
    • The first 100 KB of the log file will be used as the task log.
      • A banner information is added to the begin of log output:
        • LOG OUTPUT EXCEEDS MAXIMUM SIZE OF <log_maximum_size> MB AND IS TRUNCATED TO 100 KB.
      • The truncated task log will be compressed and added to the database.
    • The original log file will be removed.
• The History Service will consider similar behavior when performing switch-over and fail-over (compressing and storing current logs to the database).
• The threshold settings can be modified from the JOC Cockpit Settings page.

Workaround

• The attached patch can be applied to prevent an OutOfMemoryError exception with JOC Cockpit. This patch does not prevent log files of an unwanted size to be zipped and added to the database which is a feature that is provided with the next maintenance release.
• Apply the following patch for JOC Cockpit 2.1.1-2.4.0:
  • https://download.sos-berlin.com/patches/patch_20220811_JOC-1358_OutOfMemoryError_on_2.1.1_2.4.0.zip
    • Please refer to the documentation JS7 - Patches for JOC Cockpit

Test Instructions

• Test 1
  • Settings. Section joc:
    • Set log_applicable_size to 1.
    • Set log_maximum_size to 2.
    • Check the logs/service-history.log log file:
      • a message about the changed settings should appear after about 1 minute.
  • Configure a workflow with the following job stdout:
    • < log_applicable_size
      • Expected result - the task log should be stored completely.
    • > log_applicable_size and < log_maximum_size
      • Expected result - the task log should be stored truncated (see Issue description: Setting for applicable size of a task log).
    • > log_maximum_size
      • Expected result - the task log should be stored truncated (see Issue description: Setting for maximum size of a task log).
    • check the logs/joc.log and logs/service-history.log log files:
      • it should contain a warning message about truncated logs.
• Test 2
  • Repeat the Test 1, but use default values for log_applicable_size (500) and log_maximum_size (1000).