Power outage
Incident Report for NTP Pool
Resolved
This incident has been resolved.
Posted Dec 20, 2020 - 01:00 UTC
Update
This has was all settled earlier; no data was lost but monitoring wasn't functional for a number of hours.
Posted Dec 19, 2020 - 23:00 UTC
Identified
Equinix Metal had a power outage (https://status.equinixmetal.com/incidents/pfgmgy1fnjcp) in the data center where the monitoring and web servers are hosted. One too many of our servers went down making the distributed file system (barely) unavailable. Power is back and the servers should recover soon.

When it comes up we'll review if we can increase replication for the critical services.

DNS service was unaffected other than having delayed updates from the monitoring system.
Posted Dec 18, 2020 - 01:15 UTC
This incident affected: Management Portal, Monitoring daemon (Newark, NJ), and DNS updates.