AlertOps Delivery Thread Cancellation Issue

Incident Report for AlertOps

Postmortem

On June 4, AlertOps experienced a system-wide issue causing alert escalations to persist even after alerts were assigned or closed. As a result, users continued to receive escalation notifications despite taking action on those alerts.

This issue was caused by a code regression that disrupted the logic responsible for halting escalation threads when alerts are updated with a resolved or assigned state.

Posted Jun 04, 2025 - 15:58 CDT

Resolved

During the impacted window, alerts that should have exited their escalation workflows continued notifying users up the escalation chain—even after being resolved—causing unnecessary alert noise and operational disruption.
Posted Jun 04, 2025 - 02:30 CDT