Some customers experiencing callout and notifications delays
Incident Report for Zuora

Timeframe: 6:05am PDT November 1, 2017 - 5:00am PDT November 2, 2017

Affected Systems: US Production

Symptoms: Delays in processing callouts, e-mails, bill runs, and notifications

Root Cause: Large volume of mixed workloads caused unexpected resource contention, resulting in an insufficient dequeue rate of work requests in Zuora message queuing infrastructure, resulting in processing delays.

Resolution: Workloads were segregated and prioritized to reduce contention, relieve backlog, and ensure processing within the SLA.

Future Preventative Measures: A plan is in place to ensure that all workloads are processed and managed within their specified performance SLAs, with limited resource contention.

Posted 11 days ago. Nov 07, 2017 - 13:05 PST

Resolved
This issue is resolved.
Posted 16 days ago. Nov 02, 2017 - 18:54 PDT
Update
Production backlog affecting emails, notifications and callouts remain clear. We continue to monitor the platform for optimal performance.
Posted 16 days ago. Nov 02, 2017 - 08:12 PDT
Update
Production backlog affecting emails, notifications and callouts has been cleared. We are monitoring the platform to ensure optimal performance and in parallel we are conducting full root cause analysis investigation.
Posted 17 days ago. Nov 01, 2017 - 23:03 PDT
Update
Queues continue to resolve and latency expected to begin to improve dramatically by 7:00PM PT"
Posted 17 days ago. Nov 01, 2017 - 18:14 PDT
Update
The delays are still being investigated at this time.
Posted 17 days ago. Nov 01, 2017 - 14:44 PDT
Update
We are still monitoring the backlog traffic.
Posted 17 days ago. Nov 01, 2017 - 12:53 PDT
Monitoring
Delays steadily decreasing until normal over the next 2.5hr
Posted 17 days ago. Nov 01, 2017 - 10:07 PDT
Identified
We've identified the issue and managed to reduce the backlog of callouts and notifications.

The next update will be provided in an hour.
Posted 17 days ago. Nov 01, 2017 - 09:41 PDT
Update
We are still investigating the delays which are occurring for callouts and notifications.

The next update will be provided in 1 hour.
Posted 17 days ago. Nov 01, 2017 - 08:34 PDT
Investigating
A subset of customers are experiencing callout and notification delays.
We have applied additional resources to process the backlog.
Posted 17 days ago. Nov 01, 2017 - 06:18 PDT
This incident affected: AMERICAS (Production API, Production Batch Operations).