We identified increased memory usage on the GOV.UK Notify API yesterday related to a change that was released on Friday.
The increased memory usage caused some instances to become unable to serve requests. When this happened they were automatically shutdown and new healthy instances created. Some requests received errors while this was happening.
We turned off the change at 5:30pm yesterday. Our monitoring graphs show memory usage has returned to normal and we anticipate no further errors.
Sorry for the inconvenience to you and your users.
The GOV.UK Notify team
Posted Mar 26, 2023 - 07:35 BST
Monitoring
We have seen some intermittent spikes in 5xx error responses from the API today.
There was a spike between 1:00am and 3:30am this morning and another smaller spike between 3:30pm and 5:30pm this afternoon.
At the moment, the API is processing requests normally and we aren't seeing elevated rates of errors.
We will continue to monitor the situation and provide another update by Monday morning.