YADE should recover/resume file transfers

Current Situation

• If TransferOptions / RetryOnConnectionError is configured and an error occurs during transfer of a file then this file has to be re-transferred completely.

Desired Behavior

• Automatically detect and resume the transfer after interruptions (e.g. connection is lost) in the current YADE session.
• Detect
  • YADE knows when a transfer was interrupted and that the error is due to a recoverable network error and will resume operation with the same parameters.
  • Detection is limited to a configurable number of retries after interrupted file transfers.
• Resume
  • YADE knows the chunks that have been transferred and is capable to continue writing chunks.
• Fail-safe operation
  • YADE detects problems of modified files and inconsistencies and can fallback to re-transfer a file.

Delimitation

• This feature is not intended to resume file transfers across multiple YADE sessions.
• This feature is limited to retry transfer operation within the current YADE session.

Maintainer Notes

• New configuration setting introduced: TargetFileOptions / ResumeFiles
• Requirements:
  • TransferOptions / RetryOnConnectionError must be enabled
  • TargetFileOptions / CompressFiles must not be enabled
• General behavior for file retransfer:
  • Resume = false (default)
    • After a connection interruption, the file transfer is restarted from the beginning
  • Resume = true
    • After a connection interruption, the transfer continues from the last successfully transferred byte offset.
    • Only the remaining part of the source file is read and transferred.
    • Resume is size-based only. YADE does not automatically perform checksum or hash validation during resume.
      • Optional integrity verification can be configured separately:
        • SourceFileOtions / CheckIntegrityHash
          • The integrity check is performed only when a corresponding .md5 file exists on the source. If no such file is present, the check is skipped.
• Protocol-specific limitation:
  • HTTP-based Fragments (e.g., AzureBlobStorageFragment, HTTP(S)Fragment, WebDAVFragment)
    • As Target: file retransfer always behaves as Resume = false, even if Resume is enabled
    • As Source: the source file is always read from the beginning, but already successfully transferred bytes are skipped
• Notes:
  • If YADE overrides a configured parameter, the effective value and the reason for the override are logged.

Test Instructions

• Introduction
  • The purpose of this test is to verify that file transfers correctly resume after connection errors when ResumeFiles is enabled.
  • Connection errors are difficult to reproduce naturally, so YADE provides simulation options to inject connectivity faults for testing purposes.
    • The connectivity faults are injected according to the configured <seconds> intervals, starting after the respective Source or Target connection has been established.
• Testing Steps
  • Configure TransferOptions
    • Enable RetryOnConnectionError to allow automatic retries on connection failures.
  • Configure TargetFileOptions
    • Set ResumeFiles = true to enable resume from the last successfully transferred byte.
  • Configure Simulation Arguments
    • YADE Standalone CLI:
      • --source-sim-conn-faults=<seconds>
      • --target-sim-conn-faults=<seconds>
      • --sim-conn-faults=<seconds> (shorthand for both Source and Target, sets the connectivity faults for Source and Target using a single argument)
    • YADE JITL-Job:
      • source_sim_conn_faults=<seconds>
      • target_sim_conn_faults=<seconds>
      • sim_conn_faults=<seconds> (shorthand for both Source and Target, sets the connectivity faults for Source and Target using a single argument)
    • Notes on <seconds>:
      • Supports integers or floats, with . or , as decimal separator.
      • Examples:
        • 1 - inject one connectivity fault on the Source/Target after 1 second
        • 0.5;2.5;3 - inject three connectivity faults on the Source/Target at 0.5s, 2.5s, and 3s
  • Tip
    • Use large files for testing so that transfers take enough time for connectivity faults to occur.
    • Adjust the <seconds> intervals in the simulation arguments to match the expected transfer/retry interval duration (e.g., 1;1;1;1 or 0.5;0.5;0.5).
      • This ensures that the simulated faults happen during the transfer rather than after it finishes.
    • Use e.g. TransferOptions/RetryCountMax=3 and RetryInterval=1s
  • Run Transfer
    • Observe that after each simulated connectivity fault, the transfer resumes correctly from the last byte.
      • YADE will log the connectivity fault injections and all retry attempts.
  • Retest using all available providers:
    • Test each provider both as Source and as Target.
      • Examples:
        • Copy Local -> SFTP
        • Copy SFTP -> Local
        • Copy SFTP -> SFTP
  • Retest using connectivity fault sumulation arguments and ResumeFiles=false