We have seen no further increases in errors since our last update and this incident is now resolved.
We saw approximately 0.2% of all requests to Notify returning an error between 9:30am and 6:00pm on Tuesday 8 November.
We believe this was a networking problem with traffic from services reaching our hosting provider AWS. We haven't been able to confirm this because the error rate returned to normal levels before we could finish our investigation.
We have put in place additional logging so that if this problem reoccurs we will have better data to confirm the cause.
Posted Nov 10, 2022 - 13:00 GMT
Monitoring
Error rates have returned to normal levels since approximately 6pm yesterday evening.
We have raised the issue with our hosting provider and will continue to monitor the situation today.
Posted Nov 09, 2022 - 11:41 GMT
Investigating
We’re investigating a small percentage of requests to Notify timing out since this morning.
This is currently affecting between 0.2% and 0.3% of all requests.
Some API requests are incorrectly returning 403 errors, while others will return a 504 exception. If you are using the web interface you may see a “Something went wrong” error page.
If you see one of these errors, you should retry your action.
We are currently investigating the cause of this.
We will continue to monitor this situation and update tomorrow morning.