Inbound text message failures

Incident Report for GOV.UK Notify

Postmortem

What happened

Services can let people reply to their text messages by asking for a virtual mobile number (VMN). VMNs are set up by one of Notify’s text message providers. Services can see and reply to the messages they receive when they sign in to Notify, or get them using the API.

On 30 June some of our users told us that they were not receiving any inbound text messages.

The cause of the problem was a mistake by one of the mobile networks while doing some server maintenance. Changes to the servers that store Notify’s VMN details affected routing of some inbound messages.

After confirming that messages were not being received we contacted our provider. They investigated and alerted the mobile phone network, who were able to fix the problem.

‌

What was the impact

Between 09:00 BST June 30 and 10:37 BST July 3, we did not receive inbound text messages sent to VMNs beginning with 073.

Inbound text messages sent to all other VMNs were not affected.

Depending on their phone type, users would have had one of the following experiences:

The message would have appeared to send but they would not be aware that in fact failed to reach us
The message would have been marked as failed to send by the phone

Unfortunately, we do not know how many inbound text messages users tried to send during this time. The phone networks were not able to resend any messages that failed.

‌

What we’re doing about it

We're working with our provider and the phone network to stop this from happening again.

They're improving their monitoring and testing to make sure that any future changes do not affect Notify. If something does go wrong, they'll be made aware of it immediately so they can fix it.

We apologise for the impact this had to your service and your users. We know that the ability to receive inbound messages is often crucial for your service.

If you have any questions, please get in touch using our support page:

https://www.notifications.service.gov.uk/support.

Posted Aug 18, 2020 - 12:49 BST

Resolved

Everything continues to run as normal. We will now switch to our usual monitoring.

We will be working with our providers over the next few days to fully understand the impact, communicate that information to you and then decide what we can do to make sure this doesn't happen again.

Thanks again for your patience. We know this had a big impact on your service and your users.

Posted Jul 03, 2020 - 10:37 BST

Update

We believe the problem is now solved by the network provider.

This morning we have manually tested the numbers affected. Before 11am roughly 50% of affected numbers were working. At roughly 11am it appears full service was restored. If you are still having any issues receiving inbound text messages then please contact us.

We will continue to monitor the situation throughout the day.

Our next steps will then be to confirm the time frame and impact during this incident. When we are confident we know what happened we will communicate that on our status page and also email it directly to those services that were affected.

Once again, we apologise for the impact this has had to your service and your users.

Posted Jul 02, 2020 - 12:01 BST

Monitoring

We've been advised by the affected mobile network that the cause of the problem with inbound text messaging has now been resolved. It is likely to take a number of hours before all inbound numbers return to full service, but I'm told by tomorrow morning all numbers should be functioning again.

We will check this first thing, and will let you know. Importantly, we'll let you know what that means for messages people attempted to send you during this disruption.

Thank you for bearing with us. I know it's rubbish, we're working hard to get things back to normal for you.

Pete
GOV.UK Notify

Posted Jul 01, 2020 - 21:45 BST

Update

The issue with some inbound text numbers failing to receive text messages is still ongoing.

We have narrowed down the range of numbers affected and have emailed those service teams.

If you have not received an email from us then your inbound text messaging should be working correctly.

The mobile network that provides the affected inbound text numbers is still trying to resolve this issue.

We will update this page as soon as anything changes and before the end of the working day.

Thank you for your patience and we are sorry for the impact this is having for your users.

Posted Jul 01, 2020 - 17:43 BST

Update

The issue with some inbound text numbers failing to receive text messages is still ongoing. We have escalated this to the phone network who provide us those inbound text numbers and they are currently trying to resolve this issue.

Apologies again and we will update you before the end of the working day.

Posted Jul 01, 2020 - 14:34 BST

Update

The issue with some inbound text numbers failing to receive text messages is still ongoing. Unfortunately our inbound text message provider has so far been unable to fix this overnight. We are currently escalating this to the mobile networks.

We're very sorry for the inconvenience and impact this has caused. We know how important the ability to receive text messages is for your service.

We will update this channel as soon as we have any news, or at the very earliest in the early afternoon to confirm the issue is still ongoing.

If you have any questions then please don't hesitate to contact us via our support page - https://www.notifications.service.gov.uk/support.

Posted Jul 01, 2020 - 10:41 BST

Update

The issue with some inbound text numbers failing to receive text messages is still ongoing. We are still waiting on an update from the phone networks, and will post a new update first thing tomorrow morning.

We're very sorry for the inconvenience and impact this has caused to your service.

Posted Jun 30, 2020 - 18:00 BST

Update

We are still seeing a problem with a proportion of our inbound text numbers failing to receive text messages. The issue appears to only be affecting certain ranges of our inbound text numbers.

This appears to be an issue at the phone network level and our text message provider is currently working on our behalf to resolve this. They have not yet been able to provide us with an estimated time in which this will be fixed.

If your service has been able to receive text messages in the past few hours then we believe you are unaffected. For services that are unable to receive text messages then it is not yet clear whether any messages that users attempt to send to you will eventually be delivered or will be lost.

We will update you as soon as we find out any further information on the impact or when we expect to have this resolved.

Apologies for the inconvenience.

Posted Jun 30, 2020 - 16:31 BST

Identified

We are aware of a problem with a proportion of our inbound text numbers failing to receive text messages.

We believe this may be a problem with our inbound text message provider and the phone networks. We are currently in contact with our provider to resolve this. We will update you with further details shortly.

Posted Jun 30, 2020 - 14:59 BST

This incident affected: Text message receiving.