Alright guys. This is just weird. Figured I would check if anyone else has seen this before…
On the Alerts tab in Netdata Cloud, all of my node name as well as the associated ChartIDs for the alarm seem to be missing random letters. Every row in those columns are missing characters, but each seem to be in different positions in the node name.
This can also be seen in the node list in the right side filter. However, some of the node names in the list are displayed properly without any missing characters. Even though the node name is as long or longer.
The characters that are omitted will be for example, the 2nd ,5th ,11th ,13th ,16th and 18th characters on the first row. The next row however will omit the 2nd ,6th ,9th ,14th ,16th ,19th and 21st. It seems to vary per row. This happens with ALL rows. All nodes (various hostnames) as well as the ChartID associated with that alert. Sometimes the ChartID is missing the end of the chart. (For example “disk_space._” instead of “disk_space._home” “disk_space._mnt” etc etc).
It appears that the majority of the time, the missing characters seem to be vowels…
Just like with the nodes though, the ChartIDs don’t always seem to be missing characters (other than the 2nd half of the chart) however, sometimes they DO. For example “wb_lg_mtrcs.excld_rqsts” which should actually be “web_log_metrics.excluded_requests”.
For the node name, it will also omit the same characters. Example: “mtrcs” for node name instead of “metrics”
The really weird part is that on both the “Nodes” and “Anomalies” tab (and associated filters), everything looks fine. No issues with the node names or any missing characters.
It really is weird and I’ve been banding my head with it for probably a month at least and hopes that maybe one of the nightly updates would fix it but no luck. These machines have been running for years with these names and there were 0 changes made to any of the nodes, configs etc for quite awhile leading up to this. Just started happening out of the blue a couple months ago maybe… It is not intermittent.
I am unable to find anywhere in the configs or the logs where the nodes are actually referenced with the incorrect/missing characters.
This happens as/for multiple users, using Google sign-in, on different browsers (Chrome, FF).
Other spaces/rooms created by and/or with access by the same users do not exhibit the same issue. Seems to only be contained to just that one space.
All nodes are running on Debian 10/11 and installed/updated from apt and running the same versions as the parent node’s info below:
#netdata -v
netdata v1.37.0-71-nightly
#netdata -W buildinfo
Version: netdata v1.37.0-71-nightly
Configure options: '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Install type: binpkg-deb
Binary architecture: x86_64
Packaging distro:
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK: YES
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: NO
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: YES
IPMI: YES
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES
Debug/Developer Features:
Trace Allocations: NO
#
Everything also seems ok in “netdatacli aclk-state”. I have opted not to publicly post that here but if needed, will obfuscate the IDs, node/host names etc.
Any ideas/suggestions/etc would be appreciated.
Thanks!