Workflow, Collect, Subscriber Portal, Marketplace Applications unavailable
Incident Report for Zuora
Postmortem

TIMEFRAME:

08/07/19 7:48 AM PST - 08/07/19 9:42 AM PST

SUMMARY:

On Aug 7th 7:48 AM PST, authorized and approved cleanup work on legacy monitoring applications was initiated in a non customer facing production environment. During the cleanup, an unexpected complication arose with a package manager that caused impairment of command and control functions. This propagated to customer facing environments and affected both the primary and backup ecosystems. Normal recovery procedures were severely impeded due to items deleted that were needed for recovery.

ROOT CAUSE
Loss of items in command and control caused the environments to become unavailable in an unexpected and unintended way.

RESOLUTION

Service was restored by first rebuilding online internal components that were needed to recover production systems. Once this was done, both clusters were restarted by recreating each of the connected nodes and recreating each cluster’s quorum. As soon as these clusters were restarted and the docker registry was online, the application servers immediately became available.

FUTURE PREVENTATIVE MEASURES:

  • Create separation of docker registry and authorization layer in command and control systems.
  • Ensure loss of items in command and control do not affect production environments.
Posted Aug 09, 2019 - 15:54 PDT

Resolved
This incident has been resolved.
Posted Aug 07, 2019 - 11:24 PDT
Monitoring
The issue has now been resolved and we are monitoring to ensure no lingering issues.
Posted Aug 07, 2019 - 09:57 PDT
Update
We are continuing to investigate this issue.
Posted Aug 07, 2019 - 09:04 PDT
Update
Our Engineering teams are continuing to investigate this issue with utmost priority.
Posted Aug 07, 2019 - 08:47 PDT
Investigating
We are currently experiencing a service disruption on our Zuora Connect service, our Engineers are investigating with the highest priority.
Posted Aug 07, 2019 - 08:10 PDT
This incident affected: MARKETPLACE (CONNECT) APPLICATIONS (Marketplace (Connect) - Production).