YADE - Performance improvement

Current Situation

• YADE takes a lot of time, especially when transferring a large number of files.
  • example - SFTP transfer of 30.000 files, ~50kb per file
    • YADE displays the first message (before the current transfer startet) about the found number of files after ~ 50 minutes and depending of the server settings the connection to
      the target/source system can be closed after this time period and the transfer will fail.

Desired Behaviour

• YADE should be optimized for better performance and to reduce the total transfer time.

Maintainer note

• example - SFTP transfer of 30.000 files, ~50kb per file after optimizing
  • the first message about found files:  after ~9 seconds
  • total transfer time:  ~ 45 minutes

Test instructions

• see Fix Version/s
• configure a Source (SFTP) fragment to transfer a large number of files (100, 1.000, 10.000, 30.000)
• check the time of the first message about found files and the total execution time
• transfer the same number of files e.g. with the WinSCP program and compare the WinSCP transfer time with the YADE transfer time
• repeat the test using the FTP(S) | SMB | HTTP | WebDAV fragments