Back to announcements
Latest update
concluded
on 2020-06-26 18:11:00

[JNE07O] Extreme weather event causing connectivity issues

Emergency
2020-06-26 17:17:00 UTC

Lighting strike at our Positive Park DC has caused some networking-rack power-protection relays to trip, leading to connectivity issues to some sites. Our engineers are investigating and we will update here on restoration.
Back to the original announcement

Updates

top

[03GZCJ] Investigation and restoration continues

Progressing
2020-06-26 18:11:00 UTC

Engineers are replacing equipment damaged by the lightning strike and carefully restoring power to tripped PDUs. Some affected services have been restored, others are on track. We will update here.
top

[QVIRTJ] Some systems restored. Remaining systems on train

Progressing
2020-06-26 19:20:00 UTC

The team are carefully restoring affected equipment and replacing any failed networking and switching units. Affected platforms are thus coming back online, and the team are working to restore the rest as quickly as possible. Further updates here.
top

[3CFLVU] Further systems now up and running; continuing to restore rest

Progressing
2020-06-26 19:59:00 UTC

All networking is back up, and individually affected platforms are being restored by the DC team. Each system is being tested meticulously as it resumes service. We will update here as we further recover from the lighting strike.
top

[9F8OG3] Majority of restoration concluding

Progressing
2020-06-26 21:46:00 UTC

We have restored the majority of the client systems affected by the earlier lightning strike. Some systems and services will still need further tending; our team will work through the night to tie up any loose ends and conclude all restorations.

If your service remains unavailable, or if you have any specific issue you'd like to bring to our attention, please mail the support team and we will make sure you receive the assistance you need.

We will provide further and final updates via email, our announcement site and our @posistatus twitter feed.

We apologise for the inconvenience this lightning strike caused. As you might appreciate. We have very sophisticated protective equipment which saved our datacentre from the brunt, but it still meant the restoration of services that were temporarily rendered unavailable via safe power isolation. The team have been working diligently and according to our rehearsed DR plans throughout the restoration and will continue to do so throughout the night and weekend until every single service has been restored.
top

[385HPW] Shared database services running

Progressing
2020-06-27 14:38:00 UTC

We are glad to confirm that our shared-server cloud database service is running. So if you are on the 'truth' platform, for example, you should have full DB services. Please mail support if not.
top

[TKR509] Shared mail services all restored

Progressing
2020-06-27 16:53:00 UTC

We are pleased to confirm the restoration of all shared mail systems. If you have been unable to send/receive emails, you should be able to do so now. Note that mails sent to you earlier may still be queuing, so expect some lag in receiving them.

[5OYT2G] Lightning-restoration concluded

Concluded
2020-06-27 20:29:00 UTC

This incident is concluded. We are in direct contact with clients who need further specific help. Our UK DR sites and Positive-US services were unaffected throughout. Thank you for your patience in this once-in-a-generation event at Positive Park!