Netdata Community

Allow configuring go.d.plugin via netdata configuration

I use NixOS and it has netdata and the go plugin, with the plugin a dependency of netdata.

In NixOS you avoid global state, and 99% of config files is generated at system build time and stored immutably in a hashed path. This allows seamless updates and having multiple versions of the same package installed simultaneously. So /etc/netdata is empty.

Now I’d like to change the go.d plugin configuration, but since the package is a dependency it’s harder to inject configuration. It is possible to do, but it’s a bit of work to create a wrapper package.

Would it instead be possible to allow configure the go.d plugin via the netdata configuration? So that there’s a single source of truth. I am thinking along the lines of passing the plugin the path to the netdata configuration and it reads that.

Thoughts?

Hi, @wmertens. That is unlikely, Netdata configuration file has a custom format…

I am thinking along the lines of passing the plugin the path to the netdata configuration and it reads that.

Correct me if I am wrong, you are saying that it would be better to have one configuration file for Netdata and all its external components (if the component is a dependency)?

Well, maybe not one file, but one source of configuration. So if the netdata config says “use the go plugin with this config file”, that’s one source with two files (or directories)

I think that is possible, netdata.conf has [plugin: go.d] section where you can set go.d.plugin configuration file using command options option.

Is that what you mean?

1 Like

Yes! Great, I’ll try that out, thanks!

1 Like

@wmertens the option is -c

	ConfDir     []string `short:"c" long:"config-dir" description:"config dir to read"`

Oh, I just realized that it is a config dir :thinking: The plugin uses the dir to look for its own config file and for all the modules (apache, etc.).

Perhaps we need to add a specific option for plugin configuration file.

Yes that would be nice, now I have to symlink all the other configurations. A bit bothersome but not a real problem of coursse.

I wonder if this is a bug? If you pass -c it will use the same dir for confDir and modulesConfDir?

So go.d.conf and the modules configurations are looked for in the same directory. Not that that’s a problem, just seems like a bug?

Not a bug, it supposed to work like that.

The thing is that we need to know only the the path to the plugin.conf and modules dir is path/go.d

I agree that if we need to make it more flexible (separate options for plugin/modules dir, etc.) the logic is a subject to change :grinning_face_with_smiling_eyes:

But on line 50 it doesn’t add name to the return value, while it does do that for the other return values …

Since it’s just YAML, it would be nice if the go.d.conf file allows specifying the module configuration in the file, merging with the existing config files.

Then I could write e.g.

modules:
  prometheus:
    jobs:
        - name: node_exporter_local
          url: 'http://127.0.0.1:9100/metrics'

So the module could be a boolean or an object, and from the default config files it reads a lot of other jobs and those get merged together.

Well, it is possible to have a file like this.

jobs:
  - name: name1
    module: apache
    url: ...

  - name: name2
    module: nginx
    url: ...

  - name: name3
    module: mysql
    dsn: ...

  - ...

For me personally it would be easier to have just one configuration file, yeah :grinning_face_with_smiling_eyes:

You have some nice ideas, i think what we need is a proposal and justification so we can discuss it. I am open to any ideas.

To me, it seems the simplest that there’s a bunch of files containing YAML that get read in order, and they are merged together. All components read the same files and they use their part of the config. So I’d switch netdata.conf to YAML too.

That way you can append array items to the default config. If you want to replace parts of the default config, you should not pass it and instead provide your version.

Configuration for modules of plugins would just be under the config for the plugin.

@ilyam8 proof that it’s a bug: this configuration works:

let
  goConf = pkgs.runCommand "goConf" { }
    ''
      mkdir -p $out
      cp ${pkgs.netdata}/lib/netdata/conf.d/go.d.conf $out
      cp ${pkgs.netdata}/lib/netdata/conf.d/go.d/* $out
      chmod +w $out/*
      # Turn off mysql collecting
      ${pkgs.yq-go}/bin/yq -Pi e '.modules.mysql = false' $out/go.d.conf
      # Turn on agent collecting
      ${pkgs.yq-go}/bin/yq -Pi e '.jobs += {"name": "agent", "url": "http://127.0.0.1:3001/metrics"}' $out/prometheus.conf
    '';
in {
  services.netdata.enable = true;
  services.netdata.config = {
    "plugin:go.d" = {
      "command options" = "-c ${goConf}";
    };
  };
}

The mysql module is turned off and stops collecting data, and the agent is detected in the prometheus metrics.

As you can see, both files are in the same directory.

That is something I personally tend to agree with. We have plans to improve our configuration/configuration management, but that is not the top 1 priority on our list now.

Changing go.d.plugin configuration loading/handling logic is more realistic.

Ok, so that would already be a help, but as you can see the config is already working the hard way, so not a priority for me any more :slight_smile:

@ilyam8 Will you change the -c handling so it uses name for the modules, or will you leave it as-is until an eventual config refactor?