EOF found in spawn pipe (Netdata fails to start)

Help
1

Verify first that your question wasn’t asked before on the forum:

As far as I can tell, yes.

Briefly explain what is the problem you are having:

Two days ago a node entered an error state and now it refuses to bring up netdata. The issue appears to me to be because netdata is loading a stock config with an EOF. I’ve read the stock config file located at /usr/lib/netdata/conf.d/stream.conf and as far as I can tell there’s not an incorrect EOF in that file?

The service status is:
Active: activating (auto-restart) (Result: core-dump) since Mon 2021-09-20 16:48:31 UTC; 18s ago
Processing: error.log…
Process: 8209 ExecStart=/usr/sbin/netdata -D $EXTRA_OPTS (code=dumped, signal=SEGV)

Error.log (All this node does is repeat this on loop:

2021-09-20 16:31:48: netdata INFO : MAIN : resources control: allowed file descriptors: soft = 1024, max = 4096
2021-09-20 16:31:48: netdata INFO : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2021-09-20 16:31:48: netdata INFO : MAIN : Adjusted netdata scheduling policy to idle (5), with priority 0.
2021-09-20 16:31:48: netdata INFO : MAIN : Running with process scheduling policy ‘idle’
2021-09-20 16:31:48: netdata INFO : MAIN : netdata started on pid 1110.
2021-09-20 16:31:48: netdata INFO : MAIN : Initializing spawn client.
2021-09-20 16:31:48: netdata INFO : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
Spawn server is up.
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_CONTAINER_OS_NAME=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_CONTAINER_OS_DETECTION=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_OS_NAME=Ubuntu
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_OS_ID=ubuntu
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_OS_ID_LIKE=debian
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION=18.04.5 LTS (Bionic Beaver)
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION_ID=18.04
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_OS_DETECTION=/etc/os-release
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_HOST_IS_K8S_NODE=false
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=4.15.0-156-generic
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER=none
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-detect-virt
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=16
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Core™ i7-10700T CPU @ 2.00GHz
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_FREQ=4500000000
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_DETECTION=lscpu
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_RAM=16547028992
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=0
2021-09-20 16:31:48: netdata INFO : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
2021-09-20 16:31:48: netdata INFO : MAIN : SQLite database /var/cache/netdata/netdata-meta.db initialization
2021-09-20 16:31:48: netdata INFO : MAIN : SQLite database initialization completed
2021-09-20 16:31:48: netdata INFO : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json, so Netdata will work with the default health configuration.
2021-09-20 16:31:48: netdata INFO : MAIN : CONFIG: cannot load user config ‘/etc/netdata/stream.conf’. Will try stock config.
EOF found in spawn pipe.
Shutting down spawn server event loop.
Shutting down spawn server loop complete.

2

It has nothing to do with stream.conf, the error comes from the function in netdata/spawn_server.c at master · netdata/netdata · GitHub
Asking someone to take a look

1 Like
3

I now have this exact same issue. Everything was working fine before…

2022-03-25 17:05:01: netdata INFO : MAIN : TIMEZONE: using strftime(): 'CDT'

2022-03-25 17:05:01: netdata INFO : MAIN : TIMEZONE: fixed as 'CDT'

2022-03-25 17:05:01: netdata INFO : MAIN : SIGNAL: Not enabling reaper

2022-03-25 17:05:01: netdata INFO : MAIN : resources control: allowed file descriptors: soft = 256, max = 9223372036854775807

2022-03-25 17:05:01: netdata ERROR : MAIN : Out-Of-Memory (OOM) score setting is not supported on this system. (errno 2, No such file or directory)

2022-03-25 17:05:01: netdata INFO : MAIN : netdata started on pid 4560.

2022-03-25 17:05:01: netdata INFO : MAIN : Initializing spawn client.

2022-03-25 17:05:01: netdata INFO : MAIN : Executing /usr/local/netdata/usr/libexec/netdata/plugins.d/system-info.sh

Spawn server is up.

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_OS_NAME=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_OS_DETECTION=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_CONTAINER_IS_OFFICIAL_IMAGE=false

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_OS_NAME=mac

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_OS_ID=macOS

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_OS_ID_LIKE=mac

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION=12.2.1

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION_ID=unknown

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_OS_DETECTION=sw_vers

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_HOST_IS_K8S_NODE=false

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Darwin

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=21.3.0

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER=unknown

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=none

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=8

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_FREQ=3700000000

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_DETECTION=nproc sysctl

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_RAM=2147483648

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_RAM_DETECTION=sysctl

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=3002231670652

2022-03-25 17:05:01: netdata INFO : MAIN : NETDATA_SYSTEM_DISK_DETECTION=df

2022-03-25 17:05:01: netdata INFO : MAIN : SQLite database /usr/local/netdata/var/cache/netdata/netdata-meta.db initialization

2022-03-25 17:05:01: netdata INFO : MAIN : SQLite database initialization completed

2022-03-25 17:05:01: netdata INFO : MAIN : Cannot open the file /usr/local/netdata/var/lib/netdata/health.silencers.json, so Netdata will work with the default health configuration.

2022-03-25 17:05:01: netdata INFO : MAIN : CONFIG: cannot load user config '/usr/local/netdata/etc/netdata/stream.conf'. Will try stock config.

EOF found in spawn pipe.

Shutting down spawn server event loop.

Shutting down spawn server loop complete.
4

Hi @gfay63 !

Can you please share more of the situation? Have you updated Netdata recently? Also, if it’s possible can you share the output of netdata -W buildinfo ?

Thanks!!

5

Hi guys, we’re getting this as well on one of our nodes. Here’s my netdata -W buildinfo

Version: netdata v1.34.1
Configure options:  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-plugin-freeipmi' '--with-bundled-protobuf' '--with-zlib' '--with-math' '--with-user=netdata' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
Install type: custom
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         NO
Libraries:
    protobuf:                YES (bundled)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  YES
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    NO
    EBPF:                    NO
    IPMI:                    YES
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES

This is on

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

What’s interesting is that this is intermittent. Suddenly it (netdata) starts working again only to have it crash some time later.

I’ve definitely have had this issue pop up before but coincidentally this instance was upgraded to the netdata version above on 12-May.

6

Hi @mjtice !

Is it possible to share an error.log from a time when the crash occurred? What was the previous version of Netdata before the update?

Thanks!

7

I’m also getting this issue on MacOS. Here is my netdata -W buildinfo output:

Version: netdata v1.34.1
Configure options:  '--disable-dependency-tracking' '--disable-silent-rules' '--prefix=/usr/local/Cellar/netdata/1.34.1' '--sysconfdir=/usr/local/etc' '--localstatedir=/usr/local/var' '--libexecdir=/usr/local/Cellar/netdata/1.34.1/libexec' '--with-math' '--with-zlib' '--enable-dbengine' '--with-user=netdata' 'UUID_LIBS=-lc' 'UUID_CFLAGS=-I/usr/include' 'CC=clang' 'CFLAGS=-I/private/tmp/netdata-20220422-87515-1b18ofa/netdata-v1.34.1/resources/judy/include' 'LDFLAGS=-L/private/tmp/netdata-20220422-87515-1b18ofa/netdata-v1.34.1/resources/judy/lib' 'CXX=clang++' 'PKG_CONFIG_PATH=/usr/local/opt/json-c/lib/pkgconfig:/usr/local/opt/libuv/lib/pkgconfig:/usr/local/opt/lz4/lib/pkgconfig:/usr/local/opt/openssl@1.1/lib/pkgconfig:/usr/local/opt/protobuf/lib/pkgconfig:/usr/local/opt/protobuf-c/lib/pkgconfig' 'PKG_CONFIG_LIBDIR=/usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/12'
Install type: custom
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    NO
    cgroup Network Tracking: NO
    CUPS:                    YES
    EBPF:                    NO
    IPMI:                    NO
    NFACCT:                  NO
    perf:                    NO
    slabinfo:                NO
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: NO
8

Hi @danesparza ! Thank you for the report.

Can you please share the error.log preferably at the time of the crash? Is it happening on startup, or after a while of usage?

Thanks!