[4PB9PK] Power issue related to last week's outage

Some PDUs have failed, possibly following strain from Friday's lightning outage. We are investigating and restoring service. Some sites are presently unavailable as we reboot and restore backup PDUs. We will update here frequently.

Updates

[101FT7] Many services restored

Many services have been restored following the PDU outages this morning related to last week's lightning strike; but we are aware that there remain some client services still unavailable and are working on each, and will update here.

[LWKO5K] PDU power incident concluded: prudent work ongoing

Thank you for your patience today as we completed emergency repairs and maintenance at Positive Park. We confirm that system restorations occurred this morning with per-system specific follow-ups directly with any affected customers through the day. The issues were, of course, related to the significant lighting strike the data centre endured on Friday evening.

In addition to enhanced Positive engineering presence on site (including company directors) our partner power specialists, the UK's leading authority on high-capacity enterprise power deployments, have also completed their detailed assessments today.

Our teams have concluded that, following that recent ground lightning strike, due diligence demands further critical works in order to restore all Positive Park rack power systems and data centre-wide UPS systems to full redundancy and capacity to our exacting specification.

This live "hot" work must be done in a planned manner and in careful stages. 

We anticipate this work will be completed over the next 7-21 days during which time there is an increased chance of power interruptions. 

We will take every precaution and care to prevent any interruption. We nonetheless feel it is prudent to warn our customers that some platform and server reboots could occur in the worst case scenario. 

We have extra engineers on 24/7 standby during this period and revised operating procedures, including advanced preparation of our on-site generator capacity and escalation processes. 

If you have specific concerns about your service please contact us about fail-over options between our sites. Positive's DR data centres and US Points of Presence are unaffected.

Renewed thanks for your patience and understanding as we address the full ramifications of this once-in-a-generation event.