Netdata crashes when any go.d metric or stream are defined

Help
1

Suggested template:

Problem/Question

Netdata crashes if any go.d metric is activated (snmp or prometheus), with kernel crash

Dec  6 12:53:12 testsrv kernel: [75146.668348] PD[go.d][635502]: segfault at 9 ip 000055b81918c37d sp 00007fd9f1ca32a0 error 4 in netdata[55b81913b000+493000]
Dec  6 12:53:12 testsrv kernel: [75146.668360] Code: 41 5c c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 54 55 53 48 85 f6 0f 84 88 00 00 00 80 3e 00 48 89 f5 0f 84 7c 00 00 00 <0f> b6 47 09 48 89 fb a8 01 75 71 49 89 d4 48 83 fa ff 74 4f f6 43
Dec  6 12:53:12 testsrv systemd[1]: Started Process Core Dump (PID 635690/UID 0).
Dec  6 12:53:12 testsrv systemd-coredump[635691]: Removed old coredump core.netdata.114.398bb8c8e33b4b409a6553cc93afa20c.508906.1701852256000000.zst.
Dec  6 12:53:14 testsrv systemd-coredump[635691]: Process 635314 (netdata) of user 114 dumped core.#012#012Found module linux-vdso.so.1 with build-id: 6e3a9bdd81353aadfcb49ebac17260e02a9e3cae#012Found module libnss_systemd.so.2 with build-id: 815bac43ba00c2469158bc9d2a3aa252bbaf9ac2#012Found module liblogin.so with build-id: 6caadbef27f449440e9799fe42811d53aacbde4b#012Found module libdigestmd5.so with build-id: 8620daa85a7a9477dc7c0a1bf90e009f70532325#012Found module libdb-5.3.so with build-id: 6dc2ccece58bc114d1ce20f521a527cc5862e054#012Found module libsasldb.so with build-id: 5c20d1a7f0b78b84ee33f4fb5b32400740c82fd9#012Found module libntlm.so with build-id: 689d8c5b4adc3be695c67cd5e5a1537a51036ab0#012Found module libanonymous.so with build-id: 999174000d096f95d366a216827d64264603498d#012Found module libplain.so with build-id: 283d70fd2cced6ddd12933a00e574e4adc0b9de0#012Found module libscram.so with build-id: 869844052821e1d5d420f6496e0e84595f03ed12#012Found module libcrammd5.so with build-id: 005215d18631d9703f1cc463b9527654d2b717b0#012Found module libicudata.so.70 with build-id: b1c2496dd0543023c7a19c961bb7f3abc818f465#012Found module libgpg-error.so.0 with build-id: 3fbec71c67bee60d8aef00697ee187079b0fb307#012Found module libmongocrypt.so.0 with build-id: 6bccbf6fa1cf1a0043c569926609a1f242b1884b#012Found module libicuuc.so.70 with build-id: bef3ff1d70aadd68aab07d858a759360c8b919ae#012Found module libresolv.so.2 with build-id: 1b4df612be2417a726b432f8f4da89fe2adba9a9#012Found module libsasl2.so.2 with build-id: b8a924f277180f8743a0c6463a379b1a4ad1aae5#012Found module libgcrypt.so.20 with build-id: 60a5e524de0ed8323edf33e9eb9127a9eee02359#012Found module libcap.so.2 with build-id: b4bf900abf14aabe12d90988ceb30888acb2bcb0#012Found module liblzma.so.5 with build-id: b85da6c48eb60a646615392559483b93617ef265#012Found module ld-linux-x86-64.so.2 with build-id: 9718d3757f00d2366056830aae09698dbd35e32c#012Found module libc.so.6 with build-id: a43bfc8428df6623cd498c9c0caeb91aec9be4f9#012Found module libgcc_s.so.1 with build-id: e3a44e0da9c6e835d293ed8fd2882b4c4a87130c#012Found module libm.so.6 with build-id: d2c7d1fdefc7a876b6017c090ccd55fb21e8d77f#012Found module libstdc++.so.6 with build-id: e37fe1a879783838de78cbc8c80621fa685d58a2#012Found module libbson-1.0.so.0 with build-id: c2f4bfd9e42462c7dfd9d24b97bceebd7cf00e9d#012Found module libmongoc-1.0.so.0 with build-id: 567b9814733917f1351bc56b39119fddff2a5790#012Found module libsnappy.so.1 with build-id: a3611293f42a2a0fd39b5ba3b6ccdb1c17f387a9#012Found module libprotobuf.so.23 with build-id: 29a9a4d371168aabb6cf32f66c59cba2eaf1b0ac#012Found module libsystemd.so.0 with build-id: e7268ae5d5222eb3c780756d071132ac49467d0b#012Found module libjson-c.so.5 with build-id: 0c95d180d888c75b2b3430df4f5e497949123e36#012Found module libzstd.so.1 with build-id: 5d9d0d946a3154a748e87e17af9d14764519237b#012Found module liblz4.so.1 with build-id: a85971851cd059f1af80d553c8e7170d42ec59a1#012Found module libuv.so.1 with build-id: ff2c8af1d41a623ee738cb5839fb10384ad1c65f#012Found module libuuid.so.1 with build-id: 64c0d0cb22fa2bdeca075a0c0418ba5ff314b220#012Found module libssl.so.3 with build-id: ce838f6c51f037b73ade040b4abd647d7ae7d62d#012Found module libcrypto.so.3 with build-id: 156e054fb88f59a4100ca7edc74a79e3908027a8#012Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f#012Found module libelf.so.1 with build-id: 0eaf2d056fb292c3da2d99fa16c13d0ec798f121#012Found module netdata with build-id: c76efc64b2b961dd6bfdabdcfd73a0e316492331#012Stack trace of thread 635502:#012#0  0x000055b81918c37d n/a (netdata + 0x9237d)#012#1  0x000055b81930dab1 n/a (netdata + 0x213ab1)#012#2  0x000055b81933808b n/a (netdata + 0x23e08b)#012#3  0x000055b81918ab26 n/a (netdata + 0x90b26)#012#4  0x000055b81918c06d n/a (netdata + 0x9206d)#012#5  0x000055b81933a870 n/a (netdata + 0x240870)#012#6  0x000055b8192fa3fc n/a (netdata + 0x2003fc)#012#7  0x000055b819304795 n/a (netdata + 0x20a795)#012#8  0x000055b8192f50ca n/a (netdata + 0x1fb0ca)#012#9  0x000055b8191c216f n/a (netdata + 0xc816f)#012#10 0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#11 0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635471:#012#0  0x00007fda0e04825b __lll_lock_wait_private (libc.so.6 + 0x9125b)#012#1  0x00007fda0e05c3c8 malloc (libc.so.6 + 0xa53c8)#012#2  0x000055b8193963bb n/a (netdata + 0x29c3bb)#012#3  0x000055b8193699b1 n/a (netdata + 0x26f9b1)#012#4  0x000055b8193b87d0 n/a (netdata + 0x2be7d0)#012#5  0x000055b8193ba245 n/a (netdata + 0x2c0245)#012#6  0x000055b8193bd909 n/a (netdata + 0x2c3909)#012#7  0x000055b8193f4cc8 n/a (netdata + 0x2facc8)#012#8  0x000055b8193fd918 n/a (netdata + 0x303918)#012#9  0x000055b819348068 n/a (netdata + 0x24e068)#012#10 0x000055b8193480f1 n/a (netdata + 0x24e0f1)#012#11 0x000055b819355c11 n/a (netdata + 0x25bc11)#012#12 0x000055b81932a457 n/a (netdata + 0x230457)#012#13 0x000055b81918ab26 n/a (netdata + 0x90b26)#012#14 0x000055b81918c06d n/a (netdata + 0x9206d)#012#15 0x000055b81932aaf6 n/a (netdata + 0x230af6)#012#16 0x000055b81932b77b n/a (netdata + 0x23177b)#012#17 0x000055b819223250 n/a (netdata + 0x129250)#012#18 0x000055b81922731e n/a (netdata + 0x12d31e)#012#19 0x000055b8191c216f n/a (netdata + 0xc816f)#012#20 0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#21 0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635465:#012#0  0x00007fda0e0dd01e epoll_wait (libc.so.6 + 0x12601e)#012#1  0x00007fda0eac94c1 n/a (libuv.so.1 + 0x254c1)#012#2  0x00007fda0eab2c88 uv_run (libuv.so.1 + 0xec88)#012#3  0x000055b81934e1c9 n/a (netdata + 0x2541c9)#012#4  0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#5  0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635314:#012#0  0x00007fda0e0cba0c read (libc.so.6 + 0x114a0c)#012#1  0x00007fda0e043c36 _IO_file_underflow (libc.so.6 + 0x8cc36)#012#2  0x00007fda0e044d96 _IO_default_uflow (libc.so.6 + 0x8dd96)#012#3  0x00007fda0e03741c _IO_getline_info (libc.so.6 + 0x8041c)#012#4  0x00007fda0e036410 _IO_fgets (libc.so.6 + 0x7f410)#012#5  0x000055b81916c74c n/a (netdata + 0x7274c)#012#6  0x000055b819158459 n/a (netdata + 0x5e459)#012#7  0x00007fda0dfe0d90 n/a (libc.so.6 + 0x29d90)#012#8  0x00007fda0dfe0e40 __libc_start_main (libc.so.6 + 0x29e40)#012#9  0x000055b81915a315 n/a (netdata + 0x60315)#012#012Stack trace of thread 635316:#012#0  0x00007fda0e0dd01e epoll_wait (libc.so.6 + 0x12601e)#012#1  0x00007fda0eac94c1 n/a (libuv.so.1 + 0x254c1)#012#2  0x00007fda0eab2c88 uv_run (libuv.so.1 + 0xec88)#012#3  0x000055b81948f35e n/a (netdata + 0x39535e)#012#4  0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#5  0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635447:#012#0  0x00007fda0e0dd01e epoll_wait (libc.so.6 + 0x12601e)#012#1  0x00007fda0eac94c1 n/a (libuv.so.1 + 0x254c1)#012#2  0x00007fda0eab2c88 uv_run (libuv.so.1 + 0xec88)#012#3  0x000055b81942f881 n/a (netdata + 0x335881)#012#4  0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#5  0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635448:#012#0  0x00007fda0e048117 n/a (libc.so.6 + 0x91117)#012#1  0x00007fda0e04aa41 pthread_cond_wait (libc.so.6 + 0x93a41)#012#2  0x00007fda0eac29ad uv_cond_wait (libuv.so.1 + 0x1e9ad)#012#3  0x00007fda0eab6086 n/a (libuv.so.1 + 0x12086)#012#4  0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#5  0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635449:#012#0  0x00007fda0e048117 n/a (libc.so.6 + 0x91117)#012#1  0x00007fda0e04aa41 pthread_cond_wait (libc.so.6 + 0x93a41)#012#2  0x00007fda0eac29ad uv_cond_wait (libuv.so.1 + 0x1e9ad)#012#3  0x00007fda0eab6086 n/a (libuv.so.1 + 0x12086)#012#4  0x00007fda0e04bac3 n/a (libc.so.6 + 0x94ac3)#012#5  0x00007fda0e0dda40 n/a (libc.so.6 + 0x126a40)#012#012Stack trace of thread 635450:#012#0  0x00007fda0e048117 n/a (libc.so.6 + 0x91117)#012#1  0x00007fda0e04aa41 pthread_cond_wait (libc.so.6 + 0x93a41)#012#2  0x00007fda0eac29ad uv_cond_wait (libuv.so.1 + 0x1e9ad)#012#3  0x00007fda0eab6086 n/a (libuv.so.1 + 0x1208

Relevant docs you followed/actions you took to solve the issue

Removed any go.d metering

Environment/Browser/Agent’s version etc

Packaging:
    Netdata Version ____________________________________________ : v1.43.0-343-nightly
    Installation Type __________________________________________ : binpkg-deb
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ :  
    Configure Options __________________________________________ :  '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /var/lib/netdata/www
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 5.15.0-89-generic
    Operating System ___________________________________________ : Ubuntu
    Operating System ID ________________________________________ : ubuntu
    Operating System ID Like ___________________________________ : debian
    Operating System Version ___________________________________ : 22.04.3 LTS (Jammy Jellyfish)
    Operating System Version ID ________________________________ : none
    Detection __________________________________________________ : /etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 1
    CPU Frequency ______________________________________________ : 2400000000
    CPU Architecture ___________________________________________ : 1447915520
    RAM Bytes __________________________________________________ : 21474836480
    Disk Capacity ______________________________________________ : x86_64
    Virtualization Technology __________________________________ : microsoft
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    map ________________________________________________________ : YES
    save _______________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Judy (high-performance dynamic arrays and hashtables) ______ : YES (bundled)
    dlib (robust machine learning toolkit) _____________________ : YES (bundled)
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libm (mathematical functions) ______________________________ : YES
    jemalloc ___________________________________________________ : NO
    TCMalloc ___________________________________________________ : NO
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
    Logs Management ____________________________________________ : YES
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

What I expected to happen

Agent to stream any declared incoming streams and measure any defined and/or found metrics locally.

Tried to completely remove/purge any netdata information from the Ubuntu 22.04_3 host and re-install it via the kickstart script, alas to the same effect.

2

Hi @mx5gr

Could you also post the output of coredumpctl debug ?

3

Also, how do you remove go.d metering ? Disable go.d. plugin?

4

Hello!

First of all, I was not clear when I wrote “disable go.d metering”. I meant not specify particular metrics, such as prometheus or snmp… i.e. leave it as-is by default, performing automatic endpoint recognition for metrics.

          PID: 637836 (netdata)
           UID: 114 (netdata)
           GID: 119 (netdata)
        Signal: 11 (SEGV)
     Timestamp: Wed 2023-12-06 12:58:06 UTC (1h 35min ago)
  Command Line: /usr/sbin/netdata -D -P /var/run/netdata/netdata.pid
    Executable: /usr/sbin/netdata
 Control Group: /system.slice/netdata.service
          Unit: netdata.service
         Slice: system.slice
       Boot ID: 398bb8c8e33b4b409a6553cc93afa20c
    Machine ID: 1143c5bcaa244310a8803baf715b2d54
      Hostname: testsrv
       Storage: /var/lib/systemd/coredump/core.netdata.114.398bb8c8e33b4b409a6553cc93afa20c.637836.1701867486000000.zst (present)
     Disk Size: 6.9M
       Message: Process 637836 (netdata) of user 114 dumped core.
                
                Found module linux-vdso.so.1 with build-id: 6e3a9bdd81353aadfcb49ebac17260e02a9e3cae
                Found module libnss_systemd.so.2 with build-id: 815bac43ba00c2469158bc9d2a3aa252bbaf9ac2
                Found module liblogin.so with build-id: 6caadbef27f449440e9799fe42811d53aacbde4b
                Found module libdigestmd5.so with build-id: 8620daa85a7a9477dc7c0a1bf90e009f70532325
                Found module libdb-5.3.so with build-id: 6dc2ccece58bc114d1ce20f521a527cc5862e054
                Found module libsasldb.so with build-id: 5c20d1a7f0b78b84ee33f4fb5b32400740c82fd9
                Found module libntlm.so with build-id: 689d8c5b4adc3be695c67cd5e5a1537a51036ab0
                Found module libanonymous.so with build-id: 999174000d096f95d366a216827d64264603498d
                Found module libplain.so with build-id: 283d70fd2cced6ddd12933a00e574e4adc0b9de0
                Found module libscram.so with build-id: 869844052821e1d5d420f6496e0e84595f03ed12
                Found module libcrammd5.so with build-id: 005215d18631d9703f1cc463b9527654d2b717b0
                Found module libicudata.so.70 with build-id: b1c2496dd0543023c7a19c961bb7f3abc818f465
                Found module libgpg-error.so.0 with build-id: 3fbec71c67bee60d8aef00697ee187079b0fb307
                Found module libmongocrypt.so.0 with build-id: 6bccbf6fa1cf1a0043c569926609a1f242b1884b
                Found module libicuuc.so.70 with build-id: bef3ff1d70aadd68aab07d858a759360c8b919ae
                Found module libresolv.so.2 with build-id: 1b4df612be2417a726b432f8f4da89fe2adba9a9
                Found module libsasl2.so.2 with build-id: b8a924f277180f8743a0c6463a379b1a4ad1aae5
                Found module libgcrypt.so.20 with build-id: 60a5e524de0ed8323edf33e9eb9127a9eee02359
                Found module libcap.so.2 with build-id: b4bf900abf14aabe12d90988ceb30888acb2bcb0
                Found module liblzma.so.5 with build-id: b85da6c48eb60a646615392559483b93617ef265
                Found module ld-linux-x86-64.so.2 with build-id: 9718d3757f00d2366056830aae09698dbd35e32c
                Found module libc.so.6 with build-id: a43bfc8428df6623cd498c9c0caeb91aec9be4f9
                Found module libgcc_s.so.1 with build-id: e3a44e0da9c6e835d293ed8fd2882b4c4a87130c
                Found module libm.so.6 with build-id: d2c7d1fdefc7a876b6017c090ccd55fb21e8d77f
                Found module libstdc++.so.6 with build-id: e37fe1a879783838de78cbc8c80621fa685d58a2
                Found module libbson-1.0.so.0 with build-id: c2f4bfd9e42462c7dfd9d24b97bceebd7cf00e9d
                Found module libmongoc-1.0.so.0 with build-id: 567b9814733917f1351bc56b39119fddff2a5790
                Found module libsnappy.so.1 with build-id: a3611293f42a2a0fd39b5ba3b6ccdb1c17f387a9
                Found module libprotobuf.so.23 with build-id: 29a9a4d371168aabb6cf32f66c59cba2eaf1b0ac
                Found module libsystemd.so.0 with build-id: e7268ae5d5222eb3c780756d071132ac49467d0b
                Found module libjson-c.so.5 with build-id: 0c95d180d888c75b2b3430df4f5e497949123e36
                Found module libzstd.so.1 with build-id: 5d9d0d946a3154a748e87e17af9d14764519237b
                Found module liblz4.so.1 with build-id: a85971851cd059f1af80d553c8e7170d42ec59a1
                Found module libuv.so.1 with build-id: ff2c8af1d41a623ee738cb5839fb10384ad1c65f
                Found module libuuid.so.1 with build-id: 64c0d0cb22fa2bdeca075a0c0418ba5ff314b220
                Found module libssl.so.3 with build-id: ce838f6c51f037b73ade040b4abd647d7ae7d62d
                Found module libcrypto.so.3 with build-id: 156e054fb88f59a4100ca7edc74a79e3908027a8
                Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f
                Found module libelf.so.1 with build-id: 0eaf2d056fb292c3da2d99fa16c13d0ec798f121
                Found module netdata with build-id: c76efc64b2b961dd6bfdabdcfd73a0e316492331
                Stack trace of thread 638024:
                #0  0x0000560bd58e337d n/a (netdata + 0x9237d)
                #1  0x0000560bd5a64ab1 n/a (netdata + 0x213ab1)
                #2  0x0000560bd5a8f08b n/a (netdata + 0x23e08b)
                #3  0x0000560bd58e1b26 n/a (netdata + 0x90b26)
                #4  0x0000560bd58e306d n/a (netdata + 0x9206d)
                #5  0x0000560bd5a91870 n/a (netdata + 0x240870)
                #6  0x0000560bd5a513fc n/a (netdata + 0x2003fc)
                #7  0x0000560bd5a5b795 n/a (netdata + 0x20a795)
                #8  0x0000560bd5a4c0ca n/a (netdata + 0x1fb0ca)
                #9  0x0000560bd591916f n/a (netdata + 0xc816f)
                #10 0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #11 0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637987:
                #0  0x00007fc94d63601e epoll_wait (libc.so.6 + 0x12601e)
                #1  0x00007fc94e0224c1 n/a (libuv.so.1 + 0x254c1)
                #2  0x00007fc94e00bc88 uv_run (libuv.so.1 + 0xec88)
                #3  0x0000560bd5aa51c9 n/a (netdata + 0x2541c9)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637836:
                #0  0x00007fc94d5fa612 pause (libc.so.6 + 0xea612)
                #1  0x0000560bd58c82e9 n/a (netdata + 0x772e9)
                #2  0x0000560bd58af491 n/a (netdata + 0x5e491)
                #3  0x00007fc94d539d90 n/a (libc.so.6 + 0x29d90)
                #4  0x00007fc94d539e40 __libc_start_main (libc.so.6 + 0x29e40)
                #5  0x0000560bd58b1315 n/a (netdata + 0x60315)
                
                Stack trace of thread 637838:
                #0  0x00007fc94d63601e epoll_wait (libc.so.6 + 0x12601e)
                #1  0x00007fc94e0224c1 n/a (libuv.so.1 + 0x254c1)
                #2  0x00007fc94e00bc88 uv_run (libuv.so.1 + 0xec88)
                #3  0x0000560bd5be635e n/a (netdata + 0x39535e)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637969:
                #0  0x00007fc94d63601e epoll_wait (libc.so.6 + 0x12601e)
                #1  0x00007fc94e0224c1 n/a (libuv.so.1 + 0x254c1)
                #2  0x00007fc94e00bc88 uv_run (libuv.so.1 + 0xec88)
                #3  0x0000560bd5b86881 n/a (netdata + 0x335881)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637970:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637971:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637972:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637973:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637974:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637975:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637976:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637977:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637978:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd5aa6bcc n/a (netdata + 0x255bcc)
                #4  0x00007fc94e00eff0 n/a (libuv.so.1 + 0x11ff0)
                #5  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #6  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637979:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637980:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637981:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637982:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637983:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637984:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637985:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x00007fc94e00f086 n/a (libuv.so.1 + 0x12086)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637986:
                #0  0x00007fc94d63601e epoll_wait (libc.so.6 + 0x12601e)
                #1  0x00007fc94e0224c1 n/a (libuv.so.1 + 0x254c1)
                #2  0x00007fc94e00bc88 uv_run (libuv.so.1 + 0xec88)
                #3  0x0000560bd5aa7f2c n/a (netdata + 0x256f2c)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637991:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5be6892 n/a (netdata + 0x395892)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637992:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd5a4bb58 n/a (netdata + 0x1fab58)
                #4  0x0000560bd591916f n/a (netdata + 0xc816f)
                #5  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #6  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637993:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd597dda4 n/a (netdata + 0x12cda4)
                #4  0x0000560bd591916f n/a (netdata + 0xc816f)
                #5  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #6  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637994:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x00007fc94d5fa5ae sleep (libc.so.6 + 0xea5ae)
                #3  0x0000560bd5a4cb4a n/a (netdata + 0x1fbb4a)
                #4  0x0000560bd591916f n/a (netdata + 0xc816f)
                #5  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #6  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637995:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd58c8a56 n/a (netdata + 0x77a56)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637996:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5bbc46a n/a (netdata + 0x36b46a)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637998:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd59143bf n/a (netdata + 0xc33bf)
                #2  0x0000560bd5bc8224 n/a (netdata + 0x377224)
                #3  0x0000560bd5bc854e n/a (netdata + 0x37754e)
                #4  0x0000560bd591916f n/a (netdata + 0xc816f)
                #5  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #6  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 637999:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd5cf028c n/a (netdata + 0x49f28c)
                #2  0x0000560bd5bcc3ad n/a (netdata + 0x37b3ad)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638000:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5a69da0 n/a (netdata + 0x218da0)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638001:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd5bb4e42 n/a (netdata + 0x363e42)
                #4  0x0000560bd591916f n/a (netdata + 0xc816f)
                #5  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #6  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638002:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x0000560bd591f483 n/a (netdata + 0xce483)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638003:
                #0  0x00007fc94d624a0c read (libc.so.6 + 0x114a0c)
                #1  0x00007fc94d59cc36 _IO_file_underflow (libc.so.6 + 0x8cc36)
                #2  0x00007fc94d59dd96 _IO_default_uflow (libc.so.6 + 0x8dd96)
                #3  0x00007fc94d59041c _IO_getline_info (libc.so.6 + 0x8041c)
                #4  0x00007fc94d58f410 _IO_fgets (libc.so.6 + 0x7f410)
                #5  0x0000560bd5c50bd0 n/a (netdata + 0x3ffbd0)
                #6  0x0000560bd591916f n/a (netdata + 0xc816f)
                #7  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #8  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638004:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5bfff45 n/a (netdata + 0x3aef45)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638005:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5c01bcd n/a (netdata + 0x3b0bcd)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638006:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5bf9bfb n/a (netdata + 0x3a8bfb)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638007:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5a4907b n/a (netdata + 0x1f807b)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638008:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd5a46dd9 n/a (netdata + 0x1f5dd9)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638009:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x0000560bd5a46b23 n/a (netdata + 0x1f5b23)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638010:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x0000560bd5a46b23 n/a (netdata + 0x1f5b23)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638011:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x0000560bd5a46b23 n/a (netdata + 0x1f5b23)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638012:
                #0  0x00007fc94d63601e epoll_wait (libc.so.6 + 0x12601e)
                #1  0x00007fc94e0224c1 n/a (libuv.so.1 + 0x254c1)
                #2  0x00007fc94e00bc88 uv_run (libuv.so.1 + 0xec88)
                #3  0x0000560bd58c9cbe n/a (netdata + 0x78cbe)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
5

…and the rest of the stacktrace, due to msg size limit…

                Stack trace of thread 638013:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd59143bf n/a (netdata + 0xc33bf)
                #2  0x0000560bd5bbb00b n/a (netdata + 0x36a00b)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638015:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5bffa28 n/a (netdata + 0x3aea28)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638016:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd5c1d6ab n/a (netdata + 0x3cc6ab)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638017:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd5a5b8c7 n/a (netdata + 0x20a8c7)
                #2  0x0000560bd5a4c0ca n/a (netdata + 0x1fb0ca)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638018:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd5a5b8c7 n/a (netdata + 0x20a8c7)
                #2  0x0000560bd5a4c0ca n/a (netdata + 0x1fb0ca)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638019:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd5a5b8c7 n/a (netdata + 0x20a8c7)
                #2  0x0000560bd5a4c0ca n/a (netdata + 0x1fb0ca)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638020:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd5a5b8c7 n/a (netdata + 0x20a8c7)
                #2  0x0000560bd5a4c0ca n/a (netdata + 0x1fb0ca)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638023:
                #0  0x00007fc94d628dbf __poll (libc.so.6 + 0x118dbf)
                #1  0x0000560bd5a5b8c7 n/a (netdata + 0x20a8c7)
                #2  0x0000560bd5a4c0ca n/a (netdata + 0x1fb0ca)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638072:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x00007fc94e01b9ad uv_cond_wait (libuv.so.1 + 0x1e9ad)
                #3  0x0000560bd5bea9de n/a (netdata + 0x3999de)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638231:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x0000560bd5bd36e8 n/a (netdata + 0x3826e8)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638232:
                #0  0x00007fc94d5a1117 n/a (libc.so.6 + 0x91117)
                #1  0x00007fc94d5a3a41 pthread_cond_wait (libc.so.6 + 0x93a41)
                #2  0x0000560bd5bd36e8 n/a (netdata + 0x3826e8)
                #3  0x0000560bd591916f n/a (netdata + 0xc816f)
                #4  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #5  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
                
                Stack trace of thread 638235:
                #0  0x00007fc94d5f57f8 clock_nanosleep (libc.so.6 + 0xe57f8)
                #1  0x00007fc94d5fa677 __nanosleep (libc.so.6 + 0xea677)
                #2  0x0000560bd58de6df n/a (netdata + 0x8d6df)
                #3  0x0000560bd58dead6 n/a (netdata + 0x8dad6)
                #4  0x0000560bd58c3a05 n/a (netdata + 0x72a05)
                #5  0x0000560bd591916f n/a (netdata + 0xc816f)
                #6  0x00007fc94d5a4ac3 n/a (libc.so.6 + 0x94ac3)
                #7  0x00007fc94d636a40 n/a (libc.so.6 + 0x126a40)
6

@mx5gr can you do the following:

  • install netdata-dbgsym
  • restart netdata, wait for it to crash.
  • do coredumpctl debug

and after all these “Stack trace of thread” when you will be able to type commands type bt. It should provide additional info about where it crashes.

7

netdata-dbgsym package was not found using apt-get under Ubuntu 22. Tried netdata-core-dbgsym as well, to no avail.

If I executed your instructions correctly, the bt output is:

Core was generated by `/usr/sbin/netdata -D -P /var/run/netdata/netdata.pid'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055d739d5b37d in ?? ()
[Current thread is 1 (Thread 0x7ff6fce18640 (LWP 657769))]
(gdb) bt
#0  0x000055d739d5b37d in ?? ()
#1  0x000055d739edcab1 in ?? ()
#2  0x000055d739f0708b in ?? ()
#3  0x000055d739d59b26 in ?? ()
#4  0x000055d739d5b06d in ?? ()
#5  0x000055d739f09870 in ?? ()
#6  0x000055d739ec93fc in ?? ()
#7  0x000055d739ed3795 in ?? ()
#8  0x000055d739ec40ca in ?? ()
#9  0x000055d739d9116f in ?? ()
#10 0x00007ff7189bdac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ff718a4fa40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
8

Can you show

dpkg -l | grep netdata
apt-cache search netdata

Deb12

my output 
 $ dpkg -l | grep netdata
ii  netdata                        1.43.0-343-nightly             amd64        real-time charts for system monitoring
ii  netdata-ebpf-code-legacy       1.43.0-343-nightly             amd64        Compiled eBPF legacy code for the Netdata eBPF plugin
ii  netdata-plugin-apps            1.43.0-343-nightly             amd64        The per-application metrics collection plugin for the Netdata Agent
ii  netdata-plugin-chartsd         1.43.0-343-nightly             all          The charts.d metrics collection plugin for the Netdata Agent
ii  netdata-plugin-debugfs         1.43.0-343-nightly             amd64        The debugfs metrics collector for the Netdata Agent
ii  netdata-plugin-ebpf            1.43.0-343-nightly             amd64        The eBPF metrics collection plugin for the Netdata Agent
ii  netdata-plugin-go              1.43.0-343-nightly             amd64        The go.d metrics collection plugin for the Netdata Agent
ii  netdata-plugin-logs-management 1.43.0-343-nightly             amd64        The logs-management plugin for the Netdata Agent
ii  netdata-plugin-nfacct          1.43.0-343-nightly             amd64        The NFACCT metrics collection plugin for the Netdata Agent
ii  netdata-plugin-perf            1.43.0-343-nightly             amd64        The perf metrics collector for the Netdata Agent
ii  netdata-plugin-pythond         1.43.0-343-nightly             all          The python.d metrics collection plugin for the Netdata Agent
ii  netdata-plugin-slabinfo        1.43.0-343-nightly             amd64        The slabinfo metrics collector for the Netdata Agent
ii  netdata-plugin-systemd-journal 1.43.0-343-nightly             amd64        The systemd-journal collector for the Netdata Agent
ii  netdata-repo-edge              2-1                            all          Configuration for the official Netdata Edge package repository.

$ apt-cache search netdata
netdata-apache2 - real-time performance monitoring (apache integration)
netdata-core - real-time performance monitoring (core)
netdata-plugins-bash - real-time performance monitoring (bash plugins)
netdata-plugins-nodejs - real-time performance monitoring (nodejs plugins)
netdata-plugins-python - real-time performance monitoring (python plugins)
netdata-web - real-time performance monitoring (web)
netdata - real-time charts for system monitoring
netdata-dbgsym - debug symbols for netdata
netdata-ebpf-code-legacy - Compiled eBPF legacy code for the Netdata eBPF plugin
netdata-plugin-apps - The per-application metrics collection plugin for the Netdata Agent
netdata-plugin-apps-dbgsym - debug symbols for netdata-plugin-apps
netdata-plugin-chartsd - The charts.d metrics collection plugin for the Netdata Agent
netdata-plugin-cups - The CUPS metrics collection plugin for the Netdata Agent
netdata-plugin-cups-dbgsym - debug symbols for netdata-plugin-cups
netdata-plugin-debugfs - The debugfs metrics collector for the Netdata Agent
netdata-plugin-debugfs-dbgsym - debug symbols for netdata-plugin-debugfs
netdata-plugin-ebpf - The eBPF metrics collection plugin for the Netdata Agent
netdata-plugin-ebpf-dbgsym - debug symbols for netdata-plugin-ebpf
netdata-plugin-freeipmi - The FreeIPMI metrics collection plugin for the Netdata Agent
netdata-plugin-freeipmi-dbgsym - debug symbols for netdata-plugin-freeipmi
netdata-plugin-go - The go.d metrics collection plugin for the Netdata Agent
netdata-plugin-logs-management - The logs-management plugin for the Netdata Agent
netdata-plugin-logs-management-dbgsym - debug symbols for netdata-plugin-logs-management
netdata-plugin-nfacct - The NFACCT metrics collection plugin for the Netdata Agent
netdata-plugin-nfacct-dbgsym - debug symbols for netdata-plugin-nfacct
netdata-plugin-perf - The perf metrics collector for the Netdata Agent
netdata-plugin-perf-dbgsym - debug symbols for netdata-plugin-perf
netdata-plugin-pythond - The python.d metrics collection plugin for the Netdata Agent
netdata-plugin-slabinfo - The slabinfo metrics collector for the Netdata Agent
netdata-plugin-slabinfo-dbgsym - debug symbols for netdata-plugin-slabinfo
netdata-plugin-systemd-journal - The systemd-journal collector for the Netdata Agent
netdata-plugin-systemd-journal-dbgsym - debug symbols for netdata-plugin-systemd-journal
netdata-repo - Configuration for the official Netdata Stable package repository.
netdata-repo-edge - Configuration for the official Netdata Edge package repository
9

Here you are…

ii  netdata                               1.44.0-2-nightly                        amd64        real-time charts for system monitoring
ii  netdata-ebpf-code-legacy              1.44.0-2-nightly                        amd64        Compiled eBPF legacy code for the Netdata eBPF plugin
ii  netdata-plugin-apps                   1.44.0-2-nightly                        amd64        The per-application metrics collection plugin for the Netdata Agent
ii  netdata-plugin-chartsd                1.44.0-2-nightly                        all          The charts.d metrics collection plugin for the Netdata Agent
ii  netdata-plugin-debugfs                1.44.0-2-nightly                        amd64        The debugfs metrics collector for the Netdata Agent
ii  netdata-plugin-ebpf                   1.44.0-2-nightly                        amd64        The eBPF metrics collection plugin for the Netdata Agent
ii  netdata-plugin-go                     1.44.0-2-nightly                        amd64        The go.d metrics collection plugin for the Netdata Agent
ii  netdata-plugin-logs-management        1.44.0-2-nightly                        amd64        The logs-management plugin for the Netdata Agent
ii  netdata-plugin-nfacct                 1.44.0-2-nightly                        amd64        The NFACCT metrics collection plugin for the Netdata Agent
ii  netdata-plugin-perf                   1.44.0-2-nightly                        amd64        The perf metrics collector for the Netdata Agent
ii  netdata-plugin-pythond                1.44.0-2-nightly                        all          The python.d metrics collection plugin for the Netdata Agent
ii  netdata-plugin-slabinfo               1.44.0-2-nightly                        amd64        The slabinfo metrics collector for the Netdata Agent
ii  netdata-plugin-systemd-journal        1.44.0-2-nightly                        amd64        The systemd-journal collector for the Netdata Agent
ii  netdata-repo-edge                     2-2                                     all          Configuration for the official Netdata Edge package repository.

and

netdata-apache2 - real-time performance monitoring (apache integration)
netdata-core - real-time performance monitoring (core)
netdata-plugins-bash - real-time performance monitoring (bash plugins)
netdata-plugins-nodejs - real-time performance monitoring (nodejs plugins)
netdata-plugins-python - real-time performance monitoring (python plugins)
netdata-web - real-time performance monitoring (web)
netdata - real-time charts for system monitoring
netdata-ebpf-code-legacy - Compiled eBPF legacy code for the Netdata eBPF plugin
netdata-plugin-apps - The per-application metrics collection plugin for the Netdata Agent
netdata-plugin-chartsd - The charts.d metrics collection plugin for the Netdata Agent
netdata-plugin-cups - The CUPS metrics collection plugin for the Netdata Agent
netdata-plugin-debugfs - The debugfs metrics collector for the Netdata Agent
netdata-plugin-ebpf - The eBPF metrics collection plugin for the Netdata Agent
netdata-plugin-freeipmi - The FreeIPMI metrics collection plugin for the Netdata Agent
netdata-plugin-go - The go.d metrics collection plugin for the Netdata Agent
netdata-plugin-logs-management - The logs-management plugin for the Netdata Agent
netdata-plugin-nfacct - The NFACCT metrics collection plugin for the Netdata Agent
netdata-plugin-perf - The perf metrics collector for the Netdata Agent
netdata-plugin-pythond - The python.d metrics collection plugin for the Netdata Agent
netdata-plugin-slabinfo - The slabinfo metrics collector for the Netdata Agent
netdata-plugin-systemd-journal - The systemd-journal collector for the Netdata Agent
netdata-repo - Configuration for the official Netdata Stable package repository.
netdata-repo-edge - Configuration for the official Netdata Edge package repository.