Connectivity issues with Americas Hosted Production and Sandbox Environments
Incident
Report for Zuora
Postmortem
TIMEFRAME:
07/Aug/20 12:23:32 PM PDT to 07/Aug/20 12:25:02 PM PDT
SUMMARY AND IMPACT:
For the affected timeframe Zuora API and Zuora UI became intermittently unavailable for our US Hosted Datacenter customers.
ROOT CAUSE:
A staged BGP routing change to mitigate the impact of a future maintenance by our US Hosted Data Center Internet Service Provider was inadvertently promoted prior to the scheduled maintenance window
RESOLUTION:
The change was immediately recognized and rolled back which stabilized this issue
FUTURE PREVENTATIVE MEASURES:
- Add additional validation steps for BGP routing changes
- Add additional change management rigor for pre-staging changes.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we’re continuing to monitor
Investigating
Connectivity issues with Americas Hosted Production and Sandbox Environments
Subject: We are investigating connectivity issues with our Americas Hosted Production and Sandbox environments. API, UI, and integrations traffic were affected in an intermittent manner. Our engineering teams are engaged and working to resolve with the highest priority.
This issue is no longer impacting. We are investigating the root cause.
Subject: We are investigating connectivity issues with our Americas Hosted Production and Sandbox environments. API, UI, and integrations traffic were affected in an intermittent manner. Our engineering teams are engaged and working to resolve with the highest priority.
This issue is no longer impacting. We are investigating the root cause.
