Update - As part of the upgrades yesterday the DNS record for www.pool.ntp.org temporarily (and mistakenly, obviously) routed around the Fastly CDN.
For the "real" traffic to the pool system that's totally fine, but there's also a regular background load of HTTP requests from slightly broken/slow clients (mostly in China) that, I am guessing, tries to set the time over HTTP. The new setup is missing a piece to "hold" the slow requests when the CDN isn't active.
There are just a couple cron jobs to be moved before the server migration is complete.
Jul 15, 19:17 UTC
Update - The website has now been migrated to the Kubernetes cluster! There are very occasional timeouts happening on IPv4 requests to the website; the cause is understood. Fix underway.
On the old servers there was a set of Varnish servers in front of the application. Most things are cached by the CDN so Varnish didn't have much to do, but it did change the request flow in subtle ways that turned out to be meaningful.
Update - https://www.pool.ntp.org/en/ is now available on IPv6 because it's not 2007 anymore and on TLS (all the internal links are non-TLS for now though). More changes coming over the next days.
Both improvements are thanks to Fastly who are providing CDN services for the websites.
Jul 13, 19:52 UTC
Update - www.pool.ntp.org is getting TLS available shortly. There was a small mishap with the sequence of updates so the site might be unavailable for some users for a few minutes.
Jul 13, 18:37 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 13, 06:06 UTC
The new cluster is running Kubernetes which makes maintenance, updates and expanding the system vastly easier. The beta system has been running on this cluster for a while.
No incidents are expected, but it is a significant (and long overdue) change to the production system.
The DNS system is not being changed in this maintenance.
Resolved -
The last 2.5 days had another outage on the DNS updates. Since the previous problem we updated/changed some of the infrastructure that distributes the updates. There was a mistake so the service didn't start up appropriately after rebooting and another mistake that this wasn't monitored properly. Both have been fixed now.
The primary impact was that DNS records weren't updated (and "rotated") during this period, causing some servers to get more traffic than usual.
Credit to Eugen Coca of Stefan cel Mare University of Suceava in Romania for reporting the unusual traffic patterns.
The monitoring has now been fixed so the configuration distribution servers are checked every minute, loud alerts fire when the DNS data is out of date and more.
Jul 13, 03:10 UTC