Website and monitoring down
Incident Report for NTP Pool
Resolved
This incident has been resolved.
Posted about 1 month ago. Aug 19, 2018 - 09:58 UTC
Monitoring
The web system has been up since about an hour ago. The monitoring system was just turned back on now.
Posted about 1 month ago. Aug 19, 2018 - 09:58 UTC
Identified
The website and monitoring system is down. Our kubernetes cluster had an internal certificate that expired today. The Tectonic install that we were using used an internal certificate for the API server that was only valid a year (and without an automated process to update it, ugh). We set back the time on the cluster to get access again and are going through the manual steps listed on https://coreos.com/tectonic/docs/latest/tls/rotate-tls.html -- ugh!

The NTP service and DNS service is operating normally.
Posted about 1 month ago. Aug 19, 2018 - 07:45 UTC
This incident affected: Management Portal, Public website, Monitoring daemon (Los Angeles), and DNS updates.