More! database trouble

Incident Report for NTP Pool System Status

Monitoring

ok, we are stable again -- hopefully.

Posted Sep 28, 2025 - 01:37 UTC

Update

The database croaked again! I'm restoring from a ~20 minute old backup.

Posted Sep 27, 2025 - 21:40 UTC

Identified

With a bit of help the kubernetes operator got the cluster working again, at least for now.

Posted Sep 26, 2025 - 01:25 UTC

Investigating

The newly reset database cluster (from last night) decided to blow itself up again. I think the extra query load and data from the new monitoring system is making the setup fragile.

Investigating.

Posted Sep 26, 2025 - 00:52 UTC

This incident affects: Management Portal, Public website, DNS updates, GeoDNS servers, Global NTP Service, and Monitoring System.