Advice on self-hosted, self-deployed, no internet, monitoring of multiple servers

Help
, , , ,
1

Hi,

Thanks for Netdata. It is great with lots of monitoring (and alerts) options. :slight_smile:

I want to see if I can deploy netdata on some virtualised servers and create a central machine to store the metrics on as well as host all visualisations. All these machines have no internet access.

What steps do I need to take to make this process as efficient as possible?

  • I need to compile/create an offline-package that I can distribute to each node (headless collector). The nodes should only stream data to a central server.
  • The package should be customised with the collectors i need.
  • I need to create an offline-package for the server that hosts the metrics database and receives the streamed data from all nodes.
  • Is a central registry the same as a streaming server? Registry | Learn Netdata Or how does this fit in with everything else? =)
  • Can the central server (parent node?) also run all alarms instead of each vm? This would make this much easier too.
  • I want to be able to update the packages, so hopefully there is a way to automate the process somewhat. (i.e. bash script that runs git clone, install-netdata.sh, etc).

Most of the machines run Alpine or Ubuntu Linux, with a few Windows servers. I can use a developement workstation with Alpine to compile netdata from sources. This works well in my testing, at least for the single host.

2

When it comes to packaging related questions those are really dependent on your distribution of choice.

As for centralized monitoring without any internet connection take a look @Streaming and replication reference | Learn Netdata

The parent netdata node then allows you to store/see charts for children as well. This is not as full featured as netdata cloud but should do.

Parent node can run alarms. You can make children headless (as you noticed already) and make only parent store the data.

  • I want to be able to update the packages, so hopefully there is a way to automate the process somewhat. (i.e. bash script that runs git clone, install-netdata.sh, etc).

This again depends on your distribution of choice and/or some admin/management solution used.
Currently netdata itself is “read-only” monitoring solution and does not provide tools for remote administration of your servers.

3

Thank you very much for the reply.

The idea I have is to follow the “install from sources” guide:

cd /mnt/tempdisk/netdata-sources
git clone https://github.com/netdata/netdata.git --depth=100 --recursive
cd netdata
export TMPDIR="/mnt/tempdisk/tmp/netdata"
export **FLAGS="<appropiate c/cxx/cpp/ld-flags>"
./netdata-installer.sh --enable-lto --enable-ml --disable-telemetry  --install-prefix /opt --dont-start-it

Can I simply zip the /opt/netdata directory and redistribute it (assuming i set up appropiate dependencies on the nodes first), and then create a systemd and openrc init scripts to launch it?

Then I wonder about the Registry | Learn Netdata registry bits. How does that fit into the overall netdata. Is this needed when I host all nodes/childrens data on the one parent?

4

Don’t think only copying the /opt/netdata will work as AFAIK we also install the systemd unit files, create netdata user etc.

What I would probably do is take a look at your distribution and how it creates the “netdata” package. Its open source right? Then modify it as you wish to build custom package?

5

There are netdata packages for both Ubuntu and Alpine, but they do not include everything that I get when I compile from sources. The systemd unit files (for ubuntu) and init script (Alpine uses openrc) are something I can manually install, as well as users. I think that is less problematic to deal with.

6

Just to note that it seems to work.

  • Compile
  • create initrd/systemd units
  • modify /etc/opt/netdata/netdata.conf
  • zip the installed files in /opt/netdata, include the initrd/systemd units
  • distribute to nodes
  • unzip, create the netdata user and add to correct groups
  • Ensure runtime-deps are installed
  • run netdata

Thanks for allowing me to discuss this =)

2 Likes
7

Glad you could find solution that works for you.

Thanks for allowing me to discuss this =)

Of course, it might help others as well who might have same needs in future

1 Like
8

I want to add that I also added a pre-populatedstream.confwith API-keys so that clients automatically sync with the parent. :slight_smile:

1 Like
9

Thank you for the advice in this thread. The setup seems to be working well with a couple annoying caveats:

  • the “parent node” is still visible as a node in the UI, despite not showing any metrics now. Does anyone know if it’s possible to disable it?
  • there are a lot of cloud features being advertised, despite me disabling cloud features
  • I guess there is only one “room” in the open source/self-hosted version?