Uploaded image for project: 'JOC - JS7 Operations Center'
  1. JOC - JS7 Operations Center
  2. JOC-2267

History Service stops while replaying history events after JOC Cockpit fail-over

    XMLWordPrintable

Details

    • Fix
    • Status: Released (View Workflow)
    • Minor
    • Resolution: Fixed
    • 2.8.1
    • 2.8.5, 2.9.1
    • None
    • None

    Description

      Current Situation :

      In certain fail-over scenarios involving PostgreSQL, the Primary JOC Cockpit loses database connectivity, resulting in an automatic fail-over to the Standby JOC. Cockpit instance. The Standby instance becomes the active cluster member and starts the History Service.

      During recovery, the History Service replays pending history events that had been partially processed before the database outage. While replaying these events, PostgreSQL reports duplicate key violations on the unique_hosteps_ch constraint. The transaction is subsequently aborted, causing all following statements in the transaction to fail. After exceeding the configured max_stop_processing_on_errors=5 threshold, the History Service stops processing events automatically.

      This behavior prevents successful recovery of history processing after a JOC Cockpit fail-over, even though replaying already processed history events is an expected recovery scenario.

      Desired Behavior :

      After a JOC Cockpit fail-over, the History Service should successfully replay pending history events and should continue processing events without stopping due to duplicate entries, ensuring that history synchronization completes automatically.

      Test Instructions

      • Configure JOC Cockpit to use PostgreSQL.
      • Run any workflow to create the history entries.
      • Use a database tool such as pgAdmin, DBeaver, etc.
        • Table JOC_VARIABLES:
          • delete the entire row or set TEXT_VALUE=0 where the NAME value contains history<controller_id>.
            • this will reset the history for the specific controller and force it to receive the already processed event entries from this controller.
      • Restart the History Service from the dashboard.
      • Expected behavior:
        • service-history.log will show WARN/ERROR messages caused by Hibernate o.h.e.j.s.SqlExceptionHelper, but no WARN/ERROR messages from c.s.j.h.c.m.HistoryModel.
          • The HistoryModel should log:
            ...[history]...[ConstraintViolation][recovery][start] ...
            ...[history]...[ConstraintViolation]...
            ...[history]...[ConstraintViolation][recovery][completed] ...

            at INFO level without stopping the History Service.

      • Run any workflow again.
        • The history availability will confirm that the History Service is up and running.

      Attachments

        Issue Links

          Activity

            People

              re Robert Ehrlich
              ritesh-telkar Ritesh Telkar
              Ritesh Telkar Ritesh Telkar
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: