Prevent file deletion on rollback when file was already transferred by another process

Current Situation

• The following scenario:
  • The YADE selects files from a Source and transfers them to a Target.
  • When a transfer fails for a specific file, the program performs a rollback and deletes the target file, assuming it was partially or incorrectly transferred.
  • However, there is a race condition:
    • the same file might be simultaneously handled by a parallel transfer process.
    • In such a case, the parallel process may successfully complete the transfer while the original process is rolling back.
    • The rollback then deletes the file from the target, even though it was actually transferred successfully by the other process.
    • This results in data loss - a file that was correctly transferred is removed due to a failure in a different process.

Desired Behaviour

• The rollback deletion is only allowed when the current process actually changed the file.

Test Instructions

• Configure a Copy operation transfer
  • Configure a LocalSource and an SFTPFragmentRef as Target
  • Configure a Source Selection from a directory that contains two large files (~1 GB each) so that the SFTP transfer takes some time
  • Configure TransferOptions / Transactional = false
• Test preparation
  • Run a successful transfer once.
    • Goal: The target directory should contain the source files.
  • Note how long the transfer needs to transfer one file – e.g., 10 seconds
• Test using simulation options to inject connectivity faults (see https://change.sos-berlin.com/browse/YADE-405)
  • Set --target-sim-conn-faults=5 (YADE as Shell Job) or target_sim_conn_faults=5 (YADE JITL Job)
    • The connectivity faults will be injected after 5 seconds (during the transfer of the first file, while the transfer of the second file has not yet started)
  • Expected execution result:
    • The transfer of both files will fail, but with different rollback behavior:
      • 1st file – will attempt to delete the target file because during the 5 seconds before disconnection the file was partially written by the current transfer

        [1][Target][rollback][<target file path>]... - [deleteFileIfExists]net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected

        • Rollback fails because the target provider is disconnected
      • 2nd file – will skip rollback because the target file was not touched by the current transfer

        [2][Target][rollback][skipped][<target file path>] target file was not touched by the current operation

• Test without simulation options
  • Run the transfer
  • During the first file transfer, remove the second file from the source
    • This will cause a YADE exception because the second file was already selected for transfer but will not be found

      com.sos.commons.vfs.exceptions.ProviderException: [Source] ... java.nio.file.NoSuchFileException

  • Expected behavior:

    [2][Target][rollback][skipped][<target file path>][ModificationDate(UTC)=<datetime>] target file was not touched by the current operation

    • This info is displayed when the target provider is connected and the target file already exists