Installation crashes the entire machine (Ubuntu, Raspberry Pi 4)

Environment

Ubuntu Server running on a Raspberry Pi 4 1GB.

Problem/Question

Whenever I try to build Netdata on a Pi 4, it crashes the entire machine. My hunch is that it’s that Netdata always builds using 4 threads, breaks, and then leaves no time for anything else on the machine

Console output looks like the following:

/tmp/netdata-protobuf-sd4N7P/protobuf-3.17.3# env CFLAGS=-fPIC CXXFLAGS= LDFLAGS=make -j4
make all-recursive
[...]
     CXX google/protobuf/util/message_differencer.lo

INFO: task kworker/2:2:122 blocked for more than 120 seconds.
   Tainted: G C E 5.11.0-1016-raspi #17-Ubuntu

What I expected to happen

My machine stays responsive.

Hi! Welcome to netdata!

Indeed, using 4 threads might peg the machine… Is it a 4 or 8GB memory model? Can you check dmesg or journalctl if there is more info on what else might be going on ? On my 8GB Pi4, it builds without some noticeable problem…

It could be memory exhustion or perhaps a problem with the filesystem/sd card that could be causing it.

If you’re feeling a bit adventurous, can you try to manually issue the compile command inside the /tmp/netdata-protobuf-sd4N7P/protobuf-3.17.3 folder using a smaller value of -j to see if it completes ok?

Thanks!

1 Like

In this case, it’s probably memory exhaustion causing thrashing due to excessive swapping (the system in question has only 1GB of RAM per the OP).

I’m working on a PR right now to add support for passing arbitrary options to make during the install process, which will let users override the default number of jobs, though with only 1GB of RAM, I’m not sure how much that may help. protobuf is C++, which is already rather resource-intensive to compile, and on top of that it does things that result in non-trivial memory usage while building it.

The long term solution will be to use native binary packages once we merge https://github.com/netdata/netdata/pull/10769 (though currently I have no idea when that may get merged).

See Allow arbitrary options to be passed to make from netdata-installer.sh. by Ferroin · Pull Request #11479 · netdata/netdata · GitHub for a possible workaround.

We’ve now got a partial fix for this in the master branch of the agent repository. Starting from tonight’s nightly (or the next stable release if you are not using nightlies), you can set MAKEOPTS="-j1" in the environment when running the installer to force it to serialize the build process, which may enable an install to work successfully in the described scenario.