Solution: With recursive=true, the transer will be done recursively
Source and Target results in the same directory hierarchy. Directories in the target system are automatically created.
New Files
If a file already exists, no transfer should be done
Solution: With overwrite_files=false, only new files will be transfered.
Modified files are not considered. New files are identified by their file name, not by the time stamp for file modification.
File Transfer with FTP and SFTP protocols
Clean-up
After 30 days files in the target system are removed
Removal by use of a job
No files are removed from the source system
The user is responsible for removing files from the source system
Removal of files is not mapped to the target system
Operations Requirements
Workflow / Job Chain
For each transferred file an order is created
A 1:1 relationship is assumed between source file, target file and order
Depending on the type of file a transfer can include a group of files that consists e.g. of a data file and a metadate file. If a group of files has to be processed and if it is handled in a different way than individual files wil be decided on by the job chain (not by the file transfer mechanism)
No transactional operation for more than one group of files
Orders are created for effectively transferred files only
Solution: With create_orders_for_new_files=true only orders for new files will be created. See YADE-427
Please note that the job chain must start with a job that renames the parameter scheduler_file_path. See YADE-428
Order
Orders use a numeric, unique ID
Orders the parameters
scheduler_file_path: the name of the target file.
See Yade-428
scheduler_file_parent: The folder with the target file
scheduler_file_name: The name of the target file (without path)
scheduler_target_file_name: The name of the target file (without path)
scheduler_file_parent: The folder with the target file
scheduler_target_file_parent: The folder with the target file
scheduler_source_file_parent: The folder with the target file
scheduler_source_file_name: The folder with the source ile
scheduler_SOSFileOperations_file_count: The number of transfered files
Performance
Quantity Structure
For starters: 700 files
Later on an increase of this number might might be required
Transfer Speed
Parameterisation of JCraft/JSch for improved speed when using SFTP
Prioritisation
Priorities are handled on a per source system basis (per server), all files from a source system are considered to be transferred with the same priority.
A number of jobs for different source systems can be used should different priorities be required
Feature
Major