Details
-
Fix
-
Status: Released (View Workflow)
-
Minor
-
Resolution: Fixed
-
2.8.1
-
None
-
None
Description
Current Situation :
In certain fail-over scenarios involving PostgreSQL, the Primary JOC Cockpit loses database connectivity, resulting in an automatic fail-over to the Standby JOC. Cockpit instance. The Standby instance becomes the active cluster member and starts the History Service.
During recovery, the History Service replays pending history events that had been partially processed before the database outage. While replaying these events, PostgreSQL reports duplicate key violations on the unique_hosteps_ch constraint. The transaction is subsequently aborted, causing all following statements in the transaction to fail. After exceeding the configured max_stop_processing_on_errors=5 threshold, the History Service stops processing events automatically.
This behavior prevents successful recovery of history processing after a JOC Cockpit fail-over, even though replaying already processed history events is an expected recovery scenario.
Desired Behavior :
After a JOC Cockpit fail-over, the History Service should successfully replay pending history events and should continue processing events without stopping due to duplicate entries, ensuring that history synchronization completes automatically.
Test Instructions
- Configure JOC Cockpit to use PostgreSQL.
- Run any workflow to create the history entries.
- Use a database tool such as pgAdmin, DBeaver, etc.
- Table JOC_VARIABLES:
- delete the entire row or set TEXT_VALUE=0 where the NAME value contains history<controller_id>.
- this will reset the history for the specific controller and force it to receive the already processed event entries from this controller.
- delete the entire row or set TEXT_VALUE=0 where the NAME value contains history<controller_id>.
- Table JOC_VARIABLES:
- Restart the History Service from the dashboard.
- Expected behavior:
- service-history.log will show WARN/ERROR messages caused by Hibernate o.h.e.j.s.SqlExceptionHelper, but no WARN/ERROR messages from c.s.j.h.c.m.HistoryModel.
- The HistoryModel should log:
...[history]...[ConstraintViolation][recovery][start] ... ...[history]...[ConstraintViolation]... ...[history]...[ConstraintViolation][recovery][completed] ...
at INFO level without stopping the History Service.
- The HistoryModel should log:
- service-history.log will show WARN/ERROR messages caused by Hibernate o.h.e.j.s.SqlExceptionHelper, but no WARN/ERROR messages from c.s.j.h.c.m.HistoryModel.
- Run any workflow again.
- The history availability will confirm that the History Service is up and running.
Attachments
Issue Links
- relates to
-
JOC-2272 Monitoring generates false "Job runs longer than expected" alarms after JOC Cockpit fail-over
-
- Released
-