Delays to letters sent after 17 March 2022

Incident Report for GOV.UK Notify

Postmortem

What happened

On Monday 21 March 2022 our print provider experienced a failure of their colour printer. At the time they thought the problem would be quick to fix. Because of this, they did not tell us about it.

They did not have another colour printer available because they were upgrading printing equipment at their backup site.

On Wednesday 23 March, they learned that the replacement parts they had ordered to fix the colour printer had been delayed. They emailed Notify at 11:04am to tell us about the problem. We spent some time confirming the impact and deciding how to communicate the issue to our users. We put an update on our status page and sent emails directly to affected services at 5:19pm.

Our print provider was unable to print between 21 March to 24 March.

After restoring functionality to their printers, they worked overtime to clear the backlog of letters. They printed and posted all of the backlog by 31 March.

‌

What have we done so far

Our team has held an internal incident review. We’ve learned some lessons about how best to use both our status page and sending emails directly to affected users. We have run a workshop to train several other team members on how to use our status page for communications during an incident.

We’ve also spoken to our print provider and, so far, agreed these 3 actions:

They will tell us about any potential issues sooner in future, even if this means a greater risk of false alarms.
If all their colour printers are having problems, they can print our letters on one of their black and white printers instead.
Agree the necessary permissions to let our print provider use a third party to print Notify’s letters in the event of another disaster.

‌

Sorry

We’d like to apologise again for the problems this has caused you and your users. We know that letters not being delivered on time can be the difference between a user being able to use your service or not.

Over the next few weeks we will continue to put further measures in place to reduce the risk of any problems in the future.

If you have any questions or feedback for us, then please use our support form at https://www.notifications.service.gov.uk/support.

Posted Apr 14, 2022 - 16:45 BST

Resolved

This incident has been resolved.

Posted Apr 14, 2022 - 16:44 BST

Update

All delayed letters have now been sent.

Letters are now being printed and posted as normal.

We are very sorry for the problems this has caused you and your users.

We will continue to monitor our letter service to make sure there are no further problems.

Next week we will provide an update with what has happened and what we are doing to make sure this doesn't happen again.

Posted Apr 01, 2022 - 10:20 BST

Update

Our first class service continues to operate as normal. The backlog of international letters has been cleared and our international service is now operating as normal.

There is still a backlog of second class letters. The backlog contains letters sent to us:
- after 5:30pm on Tuesday 22 March
- after 5:30pm on Sunday 27 March
- on Monday 28 March
- and before 5:30pm on Tuesday 29 March

We expect these letters to be printed ready for collection tomorrow, along with any second class letters sent to us between 5:30pm yesterday (Tuesday 29 March) and 5:30pm today (Wednesday 30 March).

We will publish another update before the end of tomorrow (Thursday 31 March).

Posted Mar 30, 2022 - 16:12 BST

Update

Royal Mail have collected:

- first class letters sent to us before 5:30pm on Sunday 27 March
- second class letters sent to us before 5:30pm on Monday 21 March
- international letters sent to us before 5:30pm on Thursday 24 March

Our first class service is now back to normal.

Our print provider will continue to work through the remaining backlog of second class and international letters. We expect this to take until Wednesday 30 March, after which our second class and international services will be back to normal.

We will publish another update at the end of Wednesday or sooner if there are any significant changes.

We’re sorry again for the disruption this incident has caused to you and your users. We’ll be conducting an incident review this week. We’ll share what we’ve learnt and what we plan to change.

Posted Mar 28, 2022 - 17:32 BST

Update

Our print provider has made good progress on printing delayed letters.

The following letters have been printed and will be collected by Royal Mail this afternoon:

- first class letters sent to us before 5:30pm on Sunday 27 March
- second class letters sent to us before 5:30pm on Monday 21 March
- international letters sent to us before 5:30pm on Thursday 24 March

We will publish another update before the end of today (Monday 28 March) to confirm that Royal Mail have successfully collected these letters.

Our print provider will continue to work through the remaining backlog of second class and international letters.

Posted Mar 28, 2022 - 10:55 BST

Update

Our print provider is still having problems. Engineers are working to fix the issues.

This means it will take longer than expected to print the backlog of delayed letters. They are working through the weekend to minimise these extra delays.

They expect to dispatch first class letters on Monday 28 March.

They expect to dispatch second class letters on Tuesday 29 March and Wednesday 30 March.

All international letters sent since 17 March have now been dispatched (they go through a different process and are less affected by the technical problems).

We’ll publish another update on Monday when our provider has started dispatching the letters.

We’re very sorry for the disruption this incident will cause to you and your users.

Posted Mar 25, 2022 - 15:34 GMT

Update

Our print provider is printing the backlog of delayed letters.

We’re waiting for them to confirm whether all first class and international letters will be dispatched today.

We’ll publish another update before 4pm.

Posted Mar 25, 2022 - 12:00 GMT

Monitoring

Our print provider has fixed the problem.

They have started printing the backlog of letters and will be working through the night.

They expect to dispatch all delayed first class and international letters tomorrow, Friday 25 March.

They cannot confirm how many second class letters will be dispatched tomorrow.

Unfortunately, Royal Mail is unable to offer a collection on Saturday. This means any letters that are not ready for collection on Friday will be dispatched on Monday 28 March.

We’ll publish another update by midday on Friday 25 March.

Posted Mar 24, 2022 - 17:42 GMT

Update

GOV.UK Notify’s print provider is still unable to print and post letters but a fix is in progress.

The replacement part required has been delivered and is being installed. Once the printers are working again our provider will begin to work through the backlog of letters.

We have asked for first class letters to be prioritised.

Our print provider has asked Royal Mail to do an extra collection on Saturday. This should help reduce any further delays.

We hope that printing will resume by the end of today (Thursday 24 March). We will update you again before the end of the day.

Posted Mar 24, 2022 - 16:11 GMT

Identified

GOV.UK Notify’s print provider is currently unable to print and post letters.

One of their machines has broken down and the usual backup equipment is not available. This means there will be delays while they wait for a replacement part.

The delays affect:
- second class letters sent to us after 5:30pm on Thursday 17 March
- first class letters sent to us after 5:30pm on Sunday 20 March
- international letters sent to us after 5:30pm on Sunday 20 March

If a replacement part arrives in time, our provider hopes to resume printing tomorrow, Thursday 24 March. After repairing the fault they’ll work overtime to dispatch any delayed letters.

We’ll publish an update tomorrow.

We’re sorry for the disruption this incident will cause.

Posted Mar 23, 2022 - 17:19 GMT

This incident affected: Letter sending.