Automtating the configuration of NetData agents



  • Hi all,

    I have looked over the docs and could very well have missed it but I can’t find much on the topic on automating the configuration of the agent.

    Specifically, I’m talking about things like modifying alarms and so forth.

    I raise this because I’m not clear on how one could easily (and reliably) make changes to agent config.

    The documentation I’ve seen so far says that to make changes I need to run the edit-config script-wrapper (e.g. “sudo ./edit-config go.d/lighttpd.conf”). *
    So my first challenge is that the config file doesn’t exist until I’ve run the appropriate command for the module/plugin that I want to customise.
    So I can’t just run a script to edit the file. And even if I could the main netdata.conf file really isn’t structured well for editing via shell script etc.

    I presume I can’t just just dump my version of the config in there?
    Even if I could just dump a file over that’s really not a very elegant solution since now I have to start managing the documentation process for the config files over and above documentation of what settings I need.

    So I guess I’m asking if you guys have/intend to expose a commandline interface for configuration.
    If not then is there an easy way to automate the pre-population of the module config files?

    Bearing in mind that whatever option works best it needs to be something that doesn’t need messing with when the agent gets upgraded or when the agent get a new module/plugin.

    N.B. it would be worth updating the docs to explain why and when to use the script-wrapper. i.e. use it on first run to create the config file but does it really need to be used for subsequent edits?


  • Staff

    Hi,

    Thanks for this feedback. I am so sorry to respond that late, but we are working on our response time (and the team is super small!) 🙇♂️ .

    If you want to kickstart netdata with a pre-configured set of configuration files, you can simply create the configuration files and drop them in the correct directories before you kickstart Netdata. If you make any changes, simply restart Netdata.

    Automation could come in a form of a script which 1) installs Netdata 2) downloads/copies a set of configuration files to the correct directories (e.g via a git repository). You could run another script via chron and have it update the configuration files based on the repository.

    To better illustrate my point, you can take a look at my repository for installing Netdata in docker-compose project here. When I build the Netdata container, I copy-past the configuration files that I have already defined into the correct directories, so they will be picked up when Netdata starts.

    This is something that we have taken note on and we plan to release it in the future via Netdata cloud. Soon enough we will be releasing a centralized health and alarm panel for all the claimed nodes, so centralized configuration is not very far away.

    Thanks for the feedback on the docs, we will review it internally and reply here with any update. You can’t imagine how helpful these kinds of comments are.

    Take care 🙂


  • Staff

    Odysseas is right that you can maintain config files and copy them to your nodes in whatever way works best for you. Two more things as follow-ups.

    First, I’m working on a big change to the structure of the docs, and I made sure to add some more information about why you should use edit-config. You’re right that it only needs to be used on first run, but generally it’s cleaner for us to always recommend it to avoid confusion.

    Second, we’ll soon kick off an effort to create more content, both docs and guides, that cover deploying/configuring Netdata with configuration management tools like Ansible, Puppet, Chef, and maybe even Terraform, since that’s what our own SRE team uses. We have a lot of work to do to parse what’s possible and recommended, but there are a few community-led CM efforts you might want to check out.

    Ansible: https://github.com/mrlesmithjr/ansible-netdata
    Chef: https://github.com/samof76/netdata-cookbook
    Puppet: https://forge.puppet.com/denver/netdata



  • Thanks for getting back to me.
    That’s all fair enough. Here’s some feedback below which may or may not be useful when you do come to making the changes in this area.

    To illustrate: The issue I have with seeding config files is the same as any time you do a distro upgrade on Linux: doesn’t ssh (for example) bug you about having a newer config file that differs from yours?
    The issue with me keeping a stash of configs is that I now have to keep them up-to-date to match when you guys change them upstream.
    Now, if you guys don’t want to do a commandline interface then I get that but compromise might be to allow appended configs:

    For example:

    Suppose you guys ship pihole config in: /etc/netdata/go.d/pihole.conf
    I could add my customisations-ONLY (so not the whole config) as over-rides in: /etc/netdata/go.d/pihole.conf.custom

    Then ‘all’ you’d need to do is update the agent to read those settings in as an overlay on the conf file and update the installer to preserve those files during upgrades.

    This also means that we can split out another issue that I mentioned: namely, that some of the config files are not easy to modify via sed/awk etc.
    If we have these files with appended/over-ride settings (e.g. pihole.conf.custom) then there is no reason why the syntax needs to stay the same in those custom files.


  • Staff

    Hey @Luis-Johnstone,

    These days I have been floating the idea of using consul as a decentralized dynamic configuration management + service discovery tool + healthcheck and consul-template.

    The idea is that with every netdata service, you run consul-template daemon, which polls the consul key-value store and updates the configuration templates. This way, we have a robust and dynamic way to store configuration files. Consul has many more features, such as decentralized and distributed architecture for increased robustness.

    Ideas?


Log in to reply