We are investigating a performance degradation impacting response times for a subset of EU Production tenants

Incident Report for Zuora

Postmortem

TIMEFRAME:
24/Nov/2020 9:30 AM PT to 25/Nov/2020 4:49 AM PT

SUMMARY AND IMPACT:
For the above timeframe, a subset of customers in our EU Cloud Datacenter experienced intermittent long running API calls for the following API call types:
REST Create Order
REST Create Order Preview
REST Create Subscription
REST Create Subscription Preview

ROOT CAUSE:
Very Large Billing Preview Run on a single customer tenant contributed to higher levels of DB lock contention.

RESOLUTION:
Billing Preview Run job causing this issue was terminated

FUTURE PREVENTATIVE MEASURES:
• Reduce transaction scope for large accounts to improve performance and handling for Billing Preview Jobs
• Improve Monitoring to alert when these long running transactions are being executed
• Additional monitoring around DB Lock contention

Posted Dec 07, 2020 - 11:14 PST

Resolved

This incident has been resolved.

Posted Nov 25, 2020 - 06:00 PST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 25, 2020 - 04:57 PST

Investigating

We are investigating a performance degradation impacting response times for a subset of EU Production tenants.

Posted Nov 25, 2020 - 02:49 PST

This incident affected: EUROPE - CLOUD 1 (EU1) - *.eu.zuora.com (Production API, Production Integrations).