Power outage

Incident Report for NTP Pool

Resolved

This incident has been resolved.

Posted Dec 20, 2020 - 01:00 UTC

Update

This has was all settled earlier; no data was lost but monitoring wasn't functional for a number of hours.

Posted Dec 19, 2020 - 23:00 UTC

Identified

Equinix Metal had a power outage (https://status.equinixmetal.com/incidents/pfgmgy1fnjcp) in the data center where the monitoring and web servers are hosted. One too many of our servers went down making the distributed file system (barely) unavailable. Power is back and the servers should recover soon.

When it comes up we'll review if we can increase replication for the critical services.

DNS service was unaffected other than having delayed updates from the monitoring system.

Posted Dec 18, 2020 - 01:15 UTC

This incident affected: Management Portal and DNS updates.