MySQL went down because our Ceph storage cluster filled up. That traces back to a Ceph security upgrade we'd just finished.

The new version checks S3 upload signatures more strictly than before, and that exposed a bug in the S3 client our monitoring system uses: it doesn't sign one of the headers Ceph now requires. Its uploads got rejected, and instead of giving up it retried forever. Each upload is really several requests: the bulk data went through fine, but the small file that finalizes it kept getting rejected, so the data piled up without ever completing into anything usable. More than a terabyte of duplicate data accumulated in about half a day, more once you count replication overhead. A full Ceph cluster stops accepting writes from everything, and our database lives on the same cluster, so all three MySQL nodes crashed at the same time.

Ceph was back as soon as we'd cleaned out the orphaned uploads. MySQL took longer: two of the three nodes had fallen too far behind to rejoin automatically, and their rejoin attempts took the primary down twice more before we fixed them by hand.

The site was down a bit over two hours in total. Monitoring data from that window never got stored and can't be recovered. No other data was lost. The client library's bug is already fixed upstream, but the fix hasn't shipped in a release Mimir or Tempo use yet, so we moved that traffic to a connection where it signs everything correctly instead of waiting for it.