Between 9:55AM - 10:24AM Geneva time, the Proton API had an outage. All services are now operational and no emails or data were lost. We apologize for the inconvenience.
The root cause is a critical bug in the MySQL software published by Oracle. This software bug was also responsible for a previous incident in December: http://protonstatus.com/incidents/20#update-21 This bug occurs intermittently and can lock entire MySQL instances, but while in this state, MySQL does not gracefully exit, but will continue to accept connections which hang.
Unfortunately, it took us some time to trace the problem within the MySQL software. Upon completion of that, we proceeded to upgrade all MySQL replica slave instances in our infrastructure. Unfortunately, we still didn't yet have sufficient time to implement the fix on the MySQL master instances. This operation was actually scheduled for this week, as it was waiting for critical security/reliability patching to take place on our network equipment first. Unfortunately we had bad luck and this morning the issue reoccurred on one of the master instances which didn't yet have the patch. Upon completion of the patching this week, we believe this particular critical MySQL bug should be resolved. We are monitoring this situation closely and will also communicate with the software vendor about this issue.