Update - As part of the upgrades yesterday the DNS record for www.pool.ntp.org temporarily (and mistakenly, obviously) routed around the Fastly CDN.

For the "real" traffic to the pool system that's totally fine, but there's also a regular background load of HTTP requests from slightly broken/slow clients (mostly in China) that, I am guessing, tries to set the time over HTTP. The new setup is missing a piece to "hold" the slow requests when the CDN isn't active.

There are just a couple cron jobs to be moved before the server migration is complete.
Jul 15, 19:17 UTC
Update - The website has now been migrated to the Kubernetes cluster! There are very occasional timeouts happening on IPv4 requests to the website; the cause is understood. Fix underway.

On the old servers there was a set of Varnish servers in front of the application. Most things are cached by the CDN so Varnish didn't have much to do, but it did change the request flow in subtle ways that turned out to be meaningful.

The monitoring daemon has been upgraded to the one from https://github.com/ntppool/monitor
Jul 14, 08:17 UTC
Update - https://www.pool.ntp.org/en/ is now available on IPv6 because it's not 2007 anymore and on TLS (all the internal links are non-TLS for now though). More changes coming over the next days.

Both improvements are thanks to Fastly who are providing CDN services for the websites.
Jul 13, 19:52 UTC
Update - www.pool.ntp.org is getting TLS available shortly. There was a small mishap with the sequence of updates so the site might be unavailable for some users for a few minutes.
Jul 13, 18:37 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 13, 06:06 UTC
Scheduled - Over the next week the website and management portal will be upgraded and moved to a new cluster. (The new pages are temporarily running at www2.ntppool.org and manage2.ntppool.org

The new cluster is running Kubernetes which makes maintenance, updates and expanding the system vastly easier. The beta system has been running on this cluster for a while.

No incidents are expected, but it is a significant (and long overdue) change to the production system.

The DNS system is not being changed in this maintenance.
Management Portal   Under Maintenance
Public website   ? Under Maintenance
Monitoring daemon (Los Angeles)   Operational
DNS updates   ? Operational
GeoDNS servers   ? Operational
Global NTP Service   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Monitoring probes ?
Fetching
Active IPv4 Servers
Fetching
Active IPv6 Servers
Fetching
Past Incidents
Jul 20, 2018

No incidents reported today.

Jul 19, 2018

No incidents reported.

Jul 18, 2018

No incidents reported.

Jul 17, 2018

No incidents reported.

Jul 16, 2018

No incidents reported.

Jul 13, 2018
Resolved - The last 2.5 days had another outage on the DNS updates. Since the previous problem we updated/changed some of the infrastructure that distributes the updates. There was a mistake so the service didn't start up appropriately after rebooting and another mistake that this wasn't monitored properly. Both have been fixed now.

The primary impact was that DNS records weren't updated (and "rotated") during this period, causing some servers to get more traffic than usual.

Credit to Eugen Coca of Stefan cel Mare University of Suceava in Romania for reporting the unusual traffic patterns.

The monitoring has now been fixed so the configuration distribution servers are checked every minute, loud alerts fire when the DNS data is out of date and more.
Jul 13, 03:10 UTC
Jul 12, 2018

No incidents reported.

Jul 11, 2018

No incidents reported.

Jul 10, 2018

No incidents reported.

Jul 9, 2018

No incidents reported.

Jul 8, 2018

No incidents reported.

Jul 7, 2018

No incidents reported.

Jul 6, 2018

No incidents reported.