JobScheduler Active Cluster enqueued tasks are not started once <process_class max_process="..."/> limit is reached

Description

• When JobScheduler is running inside an active cluster and if process_class assigned to jobs has reached its max value, then the JobScheduler
  • does not start enqueued tasks although the process class is free after executing previous tasks.
  • constantly recalculates start times for all orders in distributed job chains and will not start any order.
• The problem occurs with one job being associated a process class that is shared with other jobs in two job chains.
• The problem does not occur
  • with a single JobScheduler instance
  • when using a shared process class that is associated to a number of job chains

Test Setup

• JobScheduler is setup in an active cluster with a supervisor to distribute the configurations.
• The Supervisor JobScheduler and two cluster members are updated to JobScheduler release 1.10.2

  Release	linux-x64 1.10.2
  JOC Build:	ea423a2e7a69e63b0c4de68e6e732d0c31cca9ff (2015-12-08)
  

• Two Job Chains each with single shell jobs and 5 orders with different run time are configured.
• The two shell jobs share a common process class called ssh_jobs.process_class.xml. The ssh_jobs process class has max_process=1

Steps to Reproduce

• Deploy attached testcases Ticket#2015122110000016.zip into JobScheduler Cluster
  1. Start order 01_01_repeat_every_4h in the Job_Chain1
  2. Start order 01_01_repeat_every_4h in the Job_Chain2
  3. Start order 01_05_quarterly in the Job_Chain1
  4. Start order 01_05_quarterly in the Job_Chain2

Observations

• Screenshot-1
  1. Check JobScheduler's Status banner it will show message 3 need process
  2. JobScheduler status banner shows 0 jobs running
  3. also check the Jobs tab, there the jobs will show "need process"
• Once first or second orders are completed, rest of the enqueued orders will not be started.
• JobScheduler Cluster will recalculate the next start time for the enqueued orders and reset them back to their start time.

Expected Behavior

• Once max_process limit is reach, task should be enqueued, but when process are available tasks should be started