History Service - optimize memory usage for File Transfer History

Current Situation

• When a YADE File Transfer is completed, the File Transfer History is automatically created by the History Service.
  • This process is based on a "yade_return_values" return value variable, which is a Base64-encoded string containing serialized file transfer objects, compressed using GZIP.
  • Deserialization of this string into multiple transfer objects and associated file instances happens in the main History Service thread, and the objects are then passed to asynchronous post-processing.
• Problems:
  • This approach can cause high memory usage, especially when multiple large data sets are processed at the same time, since the main process holds all the data in memory during deserialization.
  • Additionally, the post-processing currently creates threads in an uncontrolled way.
    • Each thread performs database operations, consuming connections from the JOC connection pool.
    • Processing many histories simultaneously can therefore impact the overall system behavior and database performance.

Desired Behaviour

• Process file transfer histories efficiently without consuming excessive memory.
• Handle data asynchronously with controlled parallelism, ensuring the main process remains responsive.
• Manage database operations in a way that avoids overloading connections and system resources.