<file_order_source> on JobScheduler Universal Agent

Current Situation

• The existing JobScheduler Agent does not provide the capability to create file orders from file watching.
• Using fully-featured JobScheduler instances provides that capability for the supported platforms.
• With the upcoming retirement of fully-featured JobScheduler instances for platforms other than Windows and Linux our users of platforms such as AIX, Solaris and HP-UX require some equivalent functionality for the new JobScheduler Universal Agent (JS-1291).

Desired Behavior

• The new JobScheduler Universal Agent provides a file watching capability based on the local file_order_source feature: http://www.sos-berlin.com/doc/en/scheduler/data/scheduler.file_orders.htm
• The configuration for file watching and forwarding of orders is located on the JobScheduler Master.
  • A new attribute <file_order_source remote_scheduler="..."> is provided that specifies the Agent that is in charge of watching incoming files.
• The JobScheduler Master creates orders from events for incoming files that have been forwarded by the JobScheduler Universal Agent.
  • The order identification is made up of the file name
• As a JobScheduler Universal Agent fulfills its role for any number of JobScheduler Masters therefore
  • orders will be added to any job chain that is configured for the JobScheduler Master for which the file watching is performed.

Limitations / Differences to local file orders

• When the job chain has ended processing the order, the order is put on the blacklist (because at that moment the JobScheduler Master doesn't know if the remote file has been removed or not)
  • The JobScheduler Master then asynchronously queries the JobScheduler Universal Agent if the file has been removed. If it has been removed, the order is removed from the blacklist. (So in most cases, the blacklisted state lasts only for a couple of milliseconds.)
• If a file has been triggered by JobScheduler Universal Agent, a file order has been created on the master and the file is removed before being processed then JobScheduler will not notice that this file has been removed. So the order will run, regardless of the existence of the file. The jobs have to deal with potentially missing files
• The JobScheduler Master sends a list of the files currently known to him to the Agent, so the Agent can answer as soon as a new file has to be added to this list. In order to keep this list (and the network traffic) at a reasonable size, job chains with a large number (1000s) of running orders should be avoided. This can be achieved using the max_orders attribute of a job chain.
• The regular expresions on Agent file order sources are processed with the Java regular expression libraries. These is more powerfull than the Henry Spencer regex libraries used in the JobScheduler Master.

Functional Improvements

• A file order source can be configured to wait for a group of files.
  • One file is configured as with the existing feature for a <file_order_source>. This file is moved or removed by JobScheduler according to a <file_order_sink> configuration.
  • Additional files can be configured to be waited for until the file group is complete. Additional files are not moved or removed by JobScheduler.
• Queuing for Blacklist checks: The asynchronous blacklist check (see above) might fail if the JobScheduler Universal Agent is not available when a job chain has finished. In that case the order would remain on the blacklist (even though the remote file has been removed). The blacklist checks could be queued an processed when the agent becomes available again.