Errors when posting Bill Runs for customers who have enabled parallel posting of invoices
Incident Report for Zuora
Postmortem

TIMEFRAME:
27/Jan/21 12:00 AM PST to 27/Jan/21 7:57 PM PST

SUMMARY:
Users of our “Post Invoices in Parallel” feature encountered errors for select Bill Runs executed during this time frame.

IMPACT
Production Bill Run (Batch) Operations for all Zuora data centers for users of “Post Invoices in Parallel” feature during the impact timeframe.

ROOT CAUSE:
Code regression introduced with Production Release (R290)

RESOLUTION:
• Temporarily Disable “Post Invoices in Parallel” feature to unblock customers
• Code fix applied to address original regression
• Reenable “Post Invoices in Parallel” feature

FUTURE PREVENTATIVE MEASURES:
• Additional error/exception monitoring 
• Additional QA test coverage

Posted Mar 03, 2021 - 13:25 PST

Resolved
This incident has been resolved.
Posted Jan 29, 2021 - 02:57 PST
Monitoring
A fix has been implemented and we are monitoring the results
Posted Jan 27, 2021 - 21:16 PST
Investigating
We have identified an issue where a small subset of our customers are encountering errors when posting bill runs. This affects only those who have enabled the, "Post Invoices in Parallel when Posting Bill Run," feature. Our Engineering teams have identified the cause for this and are working to get the fix implemented with highest priority.
Posted Jan 27, 2021 - 12:17 PST
This incident affected: AMERICAS - CLOUD 2 (NA2) - www|rest.zuora.com (Production Batch Operations) and AMERICAS - CLOUD 1 (NA1) - *.na.zuora.com (Production Batch Operations).