TIMEFRAME:
15/Apr/2021 3:40PM PDT to 15/Apr/2021 5:55PM PDT

SUMMARY AND IMPACT:
During the above timeline, a subset of customers in our EU Cloud datacenter experienced delays for selected asynchronous batch operations such as Data Query, Export API & AQuA API

ROOT CAUSE:
A single customer workflow triggered many concurrent Data Query jobs where the queries were not optimized for performance. This impacted our read-only database response times (latency) and caused longer than expected processing time for the impacted services.

RESOLUTION:
Once the source of the performance issue was identified, the queries causing the impact were stopped, and following a thorough review and optimization, were rerun successfully

FUTURE PREVENTATIVE MEASURES:

• Work with the customer directly to better optimize their workflow and queries
  
• Improve concurrency / rate-limiting systems for this service