GOV.UK Notify is unavailable

Incident Report for GOV.UK Notify

Postmortem

A member of the team was performing a one-off maintenance task on a non-user-facing part of our platform. As part of this work, they ran a command to delete some unneeded applications. They believed this command would only affect this non-user-facing part of our platform.

Unfortunately, this assumption was wrong and the command deleted all of our applications, including those which serve production traffic. This caused an immediate and total outage of Notify.

Once we realised the issue, we redeployed the most important applications within 15 minutes.

During the outage, users of Notify saw a ‘404: not found’ error. Both the Notify API and website were unavailable.

We have carried out a root cause analysis and identified future mitigations for such an incident. These include:

Introducing a process where developer access to production is granted for a limited time only
Make our deployment process more robust to automatically handle the case where a deleted application needs to be recreated before being deployed
Sharing details of this incident with our hosting provider, including the unexpected behaviour of the ‘delete applications’ command

If you have any questions or comments, please use our support form at https://www.notifications.service.gov.uk/support.

We’re sorry for the inconvenience to you and your users.

The GOV.UK Notify team

Posted Jul 06, 2023 - 12:07 BST

Resolved

GOV.UK Notify is working as normal

The Notify website and API were unavailable on the evening of Monday 26 June.

Notify is now working as normal. There have been no delays to sending emails, text messages or letters since 6:16pm.

The API was unavailable from 5:46pm to 5:59pm. Messages sent after 5:59pm were held in a queue until 6:16pm.

The website was unavailable from 5:46pm to 6:03pm.

Sending files by email and uploading PDF letters were unavailable from 5:46pm to 7:19pm.

Text messages received between 5:46pm and 5:59pm were delayed by up to 2 hours. We processed all received text messages.

The incident was triggered by a manual command we ran with an unexpected impact, which deleted most of our applications. We’ll be investigating and implementing changes to ensure that this issue does not happen again.

Once again, we’re very sorry for the inconvenience this incident has caused to you and your users.

We’re marking this incident as resolved. We’ll publish a postmortem within 7 working days.

If you have any questions or comments then please use our support form at https://www.notifications.service.gov.uk/support

The GOV.UK Notify team

Posted Jun 27, 2023 - 11:06 BST

Update

GOV.UK Notify is working successfully.

Emails, text messages and letters are sending with no delays.

Receiving text messages from your users during the incident may have been delayed but now should have been received by us. We are double checking this with our SMS providers and will confirm tomorrow.

If you received an error while trying to send a message then you will need to try sending the message again.

If you did not receive an error while trying to send a message, your message has been sent successfully.

We are very sorry for the inconvenience to you and your users.

If you have any questions or comments then please use our support form at https://www.notifications.service.gov.uk/support.

We will update again tomorrow morning.

Posted Jun 26, 2023 - 19:52 BST

Monitoring

Notify is running successfully again.

We are doing another review to make sure there are no remaining issues and will continue to monitor.

We will update again in the next hour.

Posted Jun 26, 2023 - 19:21 BST

Update

Notify is mostly working again. There are some parts we are continuing to fix. We are investigating:

- errors when sending files by email using our API
- errors when uploading PDF letters using our website

The Notify website and API were unavailable for about 15 minutes from 5:45pm.

If you received an error while trying to send a message then you will need to try sending the message again.

If you did not receive an error while trying to send a message, your message has been sent successfully.

There have been no delays to sending text messages or emails since 6:15pm.

We will update again in the next 30 minutes.

Posted Jun 26, 2023 - 18:46 BST

Update

We have mostly restored all functionality for GOV.UK Notify.

The GOV.UK Notify API and website are working again. We are confirming that everything has been fixed successfully.

There may be small delays in messages being sent from Notify.

We will update again in the next 30 minutes.

Posted Jun 26, 2023 - 18:12 BST

Identified

GOV.UK Notify is unavailable.

We have identified the issue and are working to fix this now.

We will update again in the next 30 minutes.

Posted Jun 26, 2023 - 17:57 BST

This incident affected: API, File uploads, Text message sending, Text message delivery receipts, Text message receiving, Email sending, Email delivery receipts, and Letter sending.