This morning our accounts associated with our netdata cloud account all started receiving hundreds of alarm notification e-mails a minute dated 2020-07-27, and even after disabling e-mail notifications in netdata cloud some are still trickling in. Strangely, we have never actually received an e-mail notification before today that I can remember. The alarm notifications all seem to be for 1m_sent_traffic_overflow
Something very strange happened to our infrastructure when we started replicating messages to another cluster today. It seems to have stopped now, but the thing is that there are email notifications that we ourselves saw delivered that should have been delivered. We’ll continue the investigation.
As to the second point, there are queues involved so it’s reasonable that a few might have trickled in even after you changed your profile (or room-level) setting. We’re implementing a redesign that among a lot of other improvements will allow such changes to affect in-flight messages as late as possible in the process, but it will take a couple of months.
Ok, so what happened is that as we tried to replicate the existing Pulsar cluster messages to a new cluster, some old messages started going back to the existing cluster again. It was noticed quickly and the replication stopped, but the damage was done. We’re still looking into the after effects, but we may currently have some wrong info on alarms and charts listed on the cloud.
1 Like