Uploaded image for project: 'YADE - Yet Another Data Exchange Tool'
  1. YADE - Yet Another Data Exchange Tool
  2. YADE-574

YADE should support the -return-values parameter to forward the file transfer history to a Master/Controller

    XMLWordPrintable

    Details

      Description

      Current Situation

      • YADE can be operated as a JITL job for JobScheduler 1.x and can be executed by its CLI.
      • When executed as a JITL job then
        • parallelism of tasks slows down performance,
        • the file transfer history is not supported if YADE is operated with an Agent that has no access to the reporting database.

      Desired Behavior

      • YADE should be executable as a shell job to provide better performance with parallel tasks.
      • When running as a JITL job then YADE should benefit from better performance by asynchronously feeding the file transfer history. This behavior can be activated by use of the parameter async_history = true.
      • The YADE shell job should be added the -return-values parameter that is assigned the temporary file used by JobScheduler to forward order variables to subsequent jobs.
        • for JobScheduler 1.13 the path of the temporary file is available from the SCHEDULER_RETURN_VALUES environment variable.
        • for JobScheduler 2.0 the path of the temporary file is available from the JS7_RETURN_VALUES environment variable.
      • If the -return-values parameter is used then YADE should write the transfer history entries (indicating which files have been transferred) in a compressed format to an order variable.

      Maintainer Notes

      • The functionality is primarily targeted towards use of YADE as a shell job with JS7.
      • For use of YADE as a JITL job with JobScheduler 1.x users can expect performance benefits.
      • It remains to be checked if the functionality later on can be made available for use of YADE with a shell job for JobScheduler 1.x
        • The file transfer history for the YADE shell job is not implemented with release 1.13.9 as this requires to pass an order variable to the temporary file indicated by SCHEDULER_RETURN_VALUES. The JobScheduler Master 1.x currently does not support to pick up order variables¬†used for the file transfer history.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                re Robert Ehrlich
                Reporter:
                re Robert Ehrlich
                Approver:
                Robert Ehrlich
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: