All Systems Operational
Management Portal Operational
Public website ? Operational
Monitoring daemon (Newark, NJ) Operational
DNS updates ? Operational
GeoDNS servers ? Operational
Global NTP Service Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Monitoring probes ?
Fetching
Active IPv4 Servers
Fetching
Active IPv6 Servers
Fetching
DNS Queries ?
Fetching
Past Incidents
May 12, 2021

No incidents reported today.

May 11, 2021

No incidents reported.

May 10, 2021

No incidents reported.

May 9, 2021
Resolved - All has been stable since we downgraded `runc`. We think it was this issue -- https://github.com/opencontainers/runc/issues/2865
May 9, 23:10 UTC
Monitoring - The system should be stable again. It took a while (to put it mildly), but we appear to have tracked the trouble down to a recent `runc` upgrade that'd sometimes get stuck when starting containers.

The trouble with had with ceph was just a side effect of this. In the process of debugging all this the ceph configuration has been made much more durable. 🀞🏻
May 9, 21:31 UTC
Identified - The system has been up and down through the day. We continue to have trouble with the Ceph system locking up. :-/
May 9, 05:25 UTC
Investigating - The ceph storage system hung this morning; the system is slowly recovering (hopefully -- it is still a bit choppy).

Monitoring isn't working, but all DNS services and the NTP service is unaffected or only minimally impacted for now.
May 8, 15:00 UTC
May 8, 2021
May 7, 2021

No incidents reported.

May 6, 2021

No incidents reported.

May 5, 2021

No incidents reported.

May 4, 2021

No incidents reported.

May 3, 2021

No incidents reported.

May 2, 2021

No incidents reported.

May 1, 2021

No incidents reported.

Apr 30, 2021

No incidents reported.

Apr 29, 2021

No incidents reported.

Apr 28, 2021

No incidents reported.