Netdata Community

Automtating the configuration of NetData agents

Hi all,

I have looked over the docs and could very well have missed it but I can’t find much on the topic on automating the configuration of the agent.

Specifically, I’m talking about things like modifying alarms and so forth.

I raise this because I’m not clear on how one could easily (and reliably) make changes to agent config.

The documentation I’ve seen so far says that to make changes I need to run the edit-config script-wrapper (e.g. “sudo ./edit-config go.d/lighttpd.conf”). *
So my first challenge is that the config file doesn’t exist until I’ve run the appropriate command for the module/plugin that I want to customise.
So I can’t just run a script to edit the file. And even if I could the main netdata.conf file really isn’t structured well for editing via shell script etc.

I presume I can’t just just dump my version of the config in there?
Even if I could just dump a file over that’s really not a very elegant solution since now I have to start managing the documentation process for the config files over and above documentation of what settings I need.

So I guess I’m asking if you guys have/intend to expose a commandline interface for configuration.
If not then is there an easy way to automate the pre-population of the module config files?

Bearing in mind that whatever option works best it needs to be something that doesn’t need messing with when the agent gets upgraded or when the agent get a new module/plugin.

N.B. it would be worth updating the docs to explain why and when to use the script-wrapper. i.e. use it on first run to create the config file but does it really need to be used for subsequent edits?

Hey @Luis-Johnstone,

These days I have been floating the idea of using consul as a decentralized dynamic configuration management + service discovery tool + healthcheck and consul-template.

The idea is that with every netdata service, you run consul-template daemon, which polls the consul key-value store and updates the configuration templates. This way, we have a robust and dynamic way to store configuration files. Consul has many more features, such as decentralized and distributed architecture for increased robustness.


Thanks for getting back to me.
That’s all fair enough. Here’s some feedback below which may or may not be useful when you do come to making the changes in this area.

To illustrate: The issue I have with seeding config files is the same as any time you do a distro upgrade on Linux: doesn’t ssh (for example) bug you about having a newer config file that differs from yours?
The issue with me keeping a stash of configs is that I now have to keep them up-to-date to match when you guys change them upstream.
Now, if you guys don’t want to do a commandline interface then I get that but compromise might be to allow appended configs:

For example:

Suppose you guys ship pihole config in: /etc/netdata/go.d/pihole.conf
I could add my customisations-ONLY (so not the whole config) as over-rides in: /etc/netdata/go.d/pihole.conf.custom

Then ‘all’ you’d need to do is update the agent to read those settings in as an overlay on the conf file and update the installer to preserve those files during upgrades.

This also means that we can split out another issue that I mentioned: namely, that some of the config files are not easy to modify via sed/awk etc.
If we have these files with appended/over-ride settings (e.g. pihole.conf.custom) then there is no reason why the syntax needs to stay the same in those custom files.

Odysseas is right that you can maintain config files and copy them to your nodes in whatever way works best for you. Two more things as follow-ups.

First, I’m working on a big change to the structure of the docs, and I made sure to add some more information about why you should use edit-config. You’re right that it only needs to be used on first run, but generally it’s cleaner for us to always recommend it to avoid confusion.

Second, we’ll soon kick off an effort to create more content, both docs and guides, that cover deploying/configuring Netdata with configuration management tools like Ansible, Puppet, Chef, and maybe even Terraform, since that’s what our own SRE team uses. We have a lot of work to do to parse what’s possible and recommended, but there are a few community-led CM efforts you might want to check out.


1 Like


Thanks for this feedback. I am so sorry to respond that late, but we are working on our response time (and the team is super small!) :bowing_man:‍♂ .

If you want to kickstart netdata with a pre-configured set of configuration files, you can simply create the configuration files and drop them in the correct directories before you kickstart Netdata. If you make any changes, simply restart Netdata.

Automation could come in a form of a script which 1) installs Netdata 2) downloads/copies a set of configuration files to the correct directories (e.g via a git repository). You could run another script via chron and have it update the configuration files based on the repository.

To better illustrate my point, you can take a look at my repository for installing Netdata in docker-compose project here. When I build the Netdata container, I copy-past the configuration files that I have already defined into the correct directories, so they will be picked up when Netdata starts.

This is something that we have taken note on and we plan to release it in the future via Netdata cloud. Soon enough we will be releasing a centralized health and alarm panel for all the claimed nodes, so centralized configuration is not very far away.

Thanks for the feedback on the docs, we will review it internally and reply here with any update. You can’t imagine how helpful these kinds of comments are.

Take care :slight_smile:

Sorry for the long delay. I’ve been playing with Ansible but it does not resolve the issue I see which is that I (or whomever manages the Ansible data files) has to now keep track of their own fork of the actual config files and their default values.
This is just asking for trouble as at some point you will deprecate some setting or add others; moreover, it’s really encouraging the cottage-industry side of DevOps where people end up doing tasks just to be doing DevOps rather than actually getting anything done. Don’t take this the wrong way of course but what I’m saying is that having to maintain my own copy of netdata config files doesn’t add value any where: we’ve simply created a necessary evil- an evil because of the time sink involved and the fact that now we’re going to inevitably end up with divergence.

Here’s what would humbly I suggest: modify the netdata “edit-config” script so that it can be invoked without user interaction using a special flag (it already takes the module as a parameter).
That way you can invoke it to create any config you want and then you can use Ansible to mangle the specific settings that you’re interested in. This also seems like a relatively small change since I appreciate that you have limited bandwidth :smiley:

OK, so I have a better solution (IMHO).

It appears that Netdata agent stores the template config files under:

So what I currently do with Ansible is:

  1. Check if conf file exists under (for example): “/etc/netdata/python.d/hddtemp.conf”
  2. If the file does not exist then copy it from:
  3. Use Ansible modules to mangle the bits of the conf that I care about in:
  4. “/etc/netdata/python.d/hddtemp.conf”

Now I don’t have to do silly things like maintain a fork of the configs just so I can modify a handful of lines in them.

What is the expected behaviour of the installer during an upgrade scenario?
Should the installer over-write existing config files under “/etc/netdata” or leave them untouched or merge new versions with installed versions under “/etc/netdata” ?
Same questions for the files under “/usr/lib/netdata/conf.d/” (same because I am curious about whether it’s worth my modifying the files under “/usr/lib/netdata/conf.d/” before I move any of them…)