Excessive load on NTP servers

Incident Report for NTP Pool

Resolved

The load appears to be just about back to normal. By server count we've almost mostly recovered from the "lost" servers during the event.

https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18/10

Posted Dec 30, 2016 - 21:22 UTC

Monitoring

Snapchat released an update to their app yesterday with a fix. They've also added some high capacity servers in some of the affected regions. 👍

Posted Dec 21, 2016 - 19:57 UTC

Update

More information:
https://news.ntppool.org/2016/12/load/
https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18

Posted Dec 20, 2016 - 03:18 UTC

Identified

The trouble in some regions of the world appears to be caused by the Snapchat application:

http://mailman.nanog.org/pipermail/nanog/2016-December/089588.html

We've tried reaching out to them.

Posted Dec 20, 2016 - 02:09 UTC

Investigating

Since around last Tuesday the NTP servers in some countries (or so it appears) have seen massively increased load. We believe it's a vendor not using the vendor system and haven't yet been able to identify who it is. It seems to mostly be affecting the United States, United Kingdom, Australia and New Zealand.

If you are running an NTP server, your service is greatly appreciated. If your server is seeing more queries than you can handle, please consider lowering the "net speed" in the server preferences rather than removing the server altogether.

If your server can handle more NTP queries than it currently gets, please consider increasing the same setting.

Posted Dec 19, 2016 - 02:16 UTC