TIMEFRAME:
2020/03/24 05:32 PM PDT to 2020/03/25 10:51 AM PDT

IMPACTED SYSTEM(S):
Customers on EU Cloud datacenter using Feature: Advanced Security Checks for Payment Pages 2.0 limiting submission rates

SUMMARY:
A Zuora UI Routing Service deployment triggered an unintended, high rate of card submission errors “Submit_Too_Quick” to take place. See: Configure Advanced Security Checks for Payment Pages 2.0 in Zuora Knowledge Center.

ROOT CAUSE:
Scheduled maintenance update to Zuora UI Routing Service in our EU Cloud datacenter executed starting at 2020/03/24 05:30 PM PDT introduced code regression.  
This regression caused the client IP value from any inbound request sent to Zuora to be written as an internal (private) IP.  This caused our internal tracking database card submissions to block traffic incorrectly based on common private IP.

RESOLUTION
Rollback of Zuora UI Routing Update was started on 10:25 AM PDT and finished by 10:51 AM PDT which resolved the issue

FUTURE PREVENTATIVE MEASURES:
• Globally exclude internal IPs from card submission rate tracking
• Additional post deployment validation check(s)
• Adding additional monitoring and error tracking capability
• Adding additional logging to improve triage capability