Sync's degraded performance issue affecting Sync's function

Incident Report for Zuora

Postmortem

Timeframe:
5/10/17 11:36 PM PST to 5/19/17 4:00 AM PST in APISandbox
5/17/17 10:49 PM PST to 5/19/17 4:35 AM PST in Production & Performance Test

Affected Systems: Production, APISandbox and Performance Test Environments

Symptoms: Intermittent failures in manual and scheduled SFDC syncs

Root Cause: A code optimization in a recent release to keep database connections alive was being triggered at incorrect intervals, causing intermittent database connectivity issues.

Resolution: Patch fixes were deployed to all affected environments to correct the code optimization in question.

Future Preventative Measures:
- Improve monitoring surrounding manual/scheduled sync failures
- Enhance synthetic transaction framework to include manual/scheduled sync transaction tests
- Additional release-level QA testing for SFDC code changes

Posted May 23, 2017 - 12:53 PDT

Resolved

We're glad to inform that this incident has now been fully resolved. We shall publish Root Cause Analysis when it becomes available.

Posted May 19, 2017 - 07:32 PDT

Monitoring

We have released a fix for this incident in all environments. Whilst we expect this incident to be fully resolved, we shall continue monitoring.

Posted May 19, 2017 - 05:38 PDT

Identified

We have managed to identify the root cause and a fix is undergoing testing before releases. We shall update again shortly.

Posted May 19, 2017 - 03:02 PDT

Investigating

Our Engineering Team is working actively on identifying the root cause.

Posted May 19, 2017 - 01:48 PDT

This incident affected: AMERICAS - CLOUD 2 (NA2) - www|rest.zuora.com (Production UI, Production API, Production Batch Operations, Production Analytics, Sandbox UI, Sandbox API, Sandbox Integrations, Sandbox Batch Operations, Sandbox Analytics) and MARKETPLACE (CONNECT) APPLICATIONS (Marketplace (Connect) - Production).