At approximately 12:00am ET on the morning of March 26, we began a normal database upgrade process. The upgrade appeared to complete normally, but near 3:00am ET we noticed load spikes on our primary database. Shortly after, it became unresponsive. We began working with our infrastructure provider and determined that the upgrade had caused corruption and our primary database was unresponsive. We then began the process of restoring our data from a replica and bringing up a new primary instance. At approximately 4:00pm ET, we completed the restoration and brought our services back online. We did not experience any data loss, and will continue to monitor our services to ensure they function normally.
Posted Mar 26, 2022 - 13:29 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 26, 2022 - 12:55 PDT
Update
We are continuing to restore the database, and will post updates as they are available.
Posted Mar 26, 2022 - 07:14 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 26, 2022 - 04:20 PDT
Identified
We are experiencing a database outage, and are working to resolve the issue.