DATE(S):

Americas Hosted Production: 2019-04-09 02:28:32 PDT - 2019-04-09 03:35:43 PDT
Europe Cloud Production: 2019-04-09 01:48:27 PDT - 2019-04-09 03:22:04 PDT
Americas Hosted Sandbox: 2019-04-08 23:53:30 PDT - 2019-04-09 03:46:55 PDT
Europe Cloud Sandbox: 2019-04-08 23:33:14 PDT - 2019-04-09 03:56:54 PDT

SUMMARY:

Between 11:33 PM PDT on April 8, 2019 and 03:57 AM PDT on April 9, 2019, Zuora’s customers across Americas and Europe Production and Sandbox environments experienced elevated error rates for transactions that involved a select few payment gateway integrations.

IMPACT:

During the impact periods mentioned above, impact was limited to Zuora’s customers who used payment gateway integrations that do not support card transactions, namely, AmazonPay, GMS, GoCardless, PayPalEC, Sony, and Wells Fargo.

ROOT CAUSES:

A scheduled release introduced a bug that prevented the system from loading the payment gateway definitions for payment gateways that do not support card transactions. This resulted in elevated error rates for payments that were processed through such payment gateways during the impact periods mentioned above.

RESOLUTION:

The affected code was reverted back to the previous version in sandbox environments and production environments, As part of the code reversion, each environment was tested for the issue mentioned in this incident and verified to be fixed. Impacted payments in payment runs for the impacted tenants were identified and re-submitted, with Customer Support working with such customers in this process.

FUTURE PREVENTATIVE MEASURES:

We have updated our unit, end-to-end, and regression tests and test automation coverage to account for payment gateway related changes that caused this incident. We have also performed a thorough review of code development, code review, testing, deployment and post-deployment monitoring processes in this area for any additional gaps and revised the guidelines for the same.