Pabau - Delays to email and SMS notifications for EU-region clinics – Incident details

Correction to our previous note: we said no messages were lost. We should not have said that — we have not verified it. During the affected windows a proportion of email and SMS dispatch attempts for EU-region clinics were refused, and whether every one of those was retried and delivered is something we are still checking. If you are an EU-region clinic and are missing an expected email or SMS from Saturday 29 August, please contact support and we will trace it for you. The service itself remains healthy and the incident stays resolved.

Resolved. Email and SMS notifications for EU-region clinics have been sending normally for over four hours with no further interruption. The cause was a single large report export that repeatedly exhausted the notification service's memory. That export has been stopped and a permanent fix is in review. Queued messages were delivered once the service recovered; no messages were lost.

Email and SMS notifications for EU-region clinics are sending normally again. The service has been stable since 19:27 London and queued messages have flowed through. We are keeping this incident open while we monitor. The underlying cause — a large report export exhausting the notification service's memory — has a fix in review that is not yet deployed, so we want a longer period of stability before calling this resolved. Next update by 22:00 London.

Update 19:25 BST: this is still ongoing. Email and SMS dispatch attempts for EU-region clinics are still being refused, and the dispatcher is now restarting roughly twice as often as at our last update. Messages sent since approximately 15:00 BST are at minimum delayed. Clinics in all other regions are unaffected. Our next update will follow on a fix, or within two hours, whichever is sooner.

Update 17:25 BST: this is still ongoing. Around two thirds of email and SMS dispatch attempts for EU-region clinics are still being refused, and messages sent since roughly 15:00 BST are at minimum delayed. The cause is a memory fault in the EU notification dispatcher, which is restarting repeatedly; a permanent fix is being prepared by our engineers. Clinics in all other regions are unaffected. We will post the next update when the fix is applied or within the next two hours, whichever is sooner.

To be precise about impact: we have confirmed that dispatch attempts are being rejected, and we are still confirming whether affected notifications are automatically re-sent once the service recovers.