Some customers experiencing callout and notifications delays
Incident
Report for Zuora
Postmortem
Timeframe: 6:05am PDT November 1, 2017 - 5:00am PDT November 2, 2017
Affected Systems: US Production
Symptoms: Delays in processing callouts, e-mails, bill runs, and notifications
Root Cause: Large volume of mixed workloads caused unexpected resource contention, resulting in an insufficient dequeue rate of work requests in Zuora message queuing infrastructure, resulting in processing delays.
Resolution: Workloads were segregated and prioritized to reduce contention, relieve backlog, and ensure processing within the SLA.
Future Preventative Measures: A plan is in place to ensure that all workloads are processed and managed within their specified performance SLAs, with limited resource contention.
Resolved
This issue is resolved.
Update
Production backlog affecting emails, notifications and callouts remain clear. We continue to monitor the platform for optimal performance.
Update
Production backlog affecting emails, notifications and callouts has been cleared. We are monitoring the platform to ensure optimal performance and in parallel we are conducting full root cause analysis investigation.
Update
Queues continue to resolve and latency expected to begin to improve dramatically by 7:00PM PT"
Update
The delays are still being investigated at this time.
Update
We are still monitoring the backlog traffic.
Monitoring
Delays steadily decreasing until normal over the next 2.5hr
Identified
We've identified the issue and managed to reduce the backlog of callouts and notifications.
The next update will be provided in an hour.
The next update will be provided in an hour.
Update
We are still investigating the delays which are occurring for callouts and notifications.
The next update will be provided in 1 hour.
The next update will be provided in 1 hour.
Investigating
A subset of customers are experiencing callout and notification delays.
We have applied additional resources to process the backlog.
We have applied additional resources to process the backlog.
This incident affected: AMERICAS - CLOUD 2 (NA2) - www|rest.zuora.com (Production API, Production Batch Operations).
