Uploaded image for project: 'YADE - Yet Another Data Exchange Tool'
  1. YADE - Yet Another Data Exchange Tool
  2. YADE-557

YADE - Performance improvement

    XMLWordPrintable

    Details

      Description

      Current Situation

      • YADE takes a lot of time, especially when transferring a large number of files.
        • example - SFTP transfer of 30.000 files, ~50kb per file
          • YADE displays the first message (before the current transfer startet) about the found number of files after ~ 50 minutes and depending of the server settings the connection to
            the target/source system can be closed after this time period and the transfer will fail.

      Desired Behaviour

      • YADE should be optimized for better performance and to reduce the total transfer time.

      Maintainer note

      • example - SFTP transfer of 30.000 files, ~50kb per file after optimizing
        • the first message about found files:  after ~9 seconds
        • total transfer time:  ~ 45 minutes

      Test instructions

      • see Fix Version/s
      • configure a Source (SFTP) fragment to transfer a large number of files (100, 1.000, 10.000, 30.000)
      • check the time of the first message about found files and the total execution time
      • transfer the same number of files e.g. with the WinSCP program and compare the WinSCP transfer time with the YADE transfer time
      • repeat the test using the FTP(S) | SMB | HTTP | WebDAV fragments

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                re Robert Ehrlich
                Reporter:
                re Robert Ehrlich
                Approver:
                Kanika Agrawal
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: