DATE(S):
2020-01-15 1:55pm PST - 2020-01-15 5:05pm PST (2020-01-15 21:55 UTC - 2020-01-16 01:05 UTC)

SUMMARY:
Bill Runs and Payment Runs failures for a subset of customers on our EU Data Center.

ROOT CAUSE:
As part of a scheduled update to address operational enhancements with scheduled jobs (Bill Run, Payment Run, Subscription Renewal, etc.), a backward incompatible change was inadvertently introduced which would stop customers scheduled job from being executed if the original user who scheduled the job was no longer active in the tenant.

RESOLUTION:
We've fixed the issue by rolling back the change and followed by a permanent patch (with testing and validation) to prevent this issue from happening again in the future.

FUTURE PREVENTATIVE MEASURES:
Although this was a one time enhancement, we are also adjusting our monitoring and alerting thresholds to be able to better detect issues related to scheduled job getting stuck.