Issues with the Proton API Monday 26th November 2018 08:56:46
The Proton API is experiencing issues. Our infrastructure team is currently investigating and will update when we have more details.
The Proton API had a large number of 500 errors (Internal Server Errors) for a roughly 15 minute period which caused issues for many users trying to use ProtonMail and ProtonVPN. The problem was caused by our primary Redis caching server crashing, which caused an automatic failover to a backup Redis server which was not able to cope with the production load, and required a manual intervention from our infrastructure team to fix.
For the time being, we have adjusted the priorities of the backup Redis servers so if the incident were to happen again, Redis will no longer fail over to the server that is too weak. Longer term, we will migrate Redis to stronger servers that can cope with increased load, and also re-architect our caching infrastructure to be more scalable.