Problem/Question
I recently changed all my servers to store more data in their dbengine. However, I was working on a server with one of my techs when they ran “apt update” and updated netdata, which extracted a blank netdata.conf over my customized one. I explained to him what he just did and to decline over-writing configs when apt upgrading, but, how would the netdata team recommend resolving this?
Relevant docs you followed/actions you took to solve the issue
Environment/Browser/Agent’s version etc
1.35.1
What I expected to happen
I’m hoping that I can move this dbengine config section into a non-default .conf file that is sourced just like netdata.conf but not overwritten on every apt upgrade. This may not exist.
If not, then, should I just “freeze” the file like so?
chattr +i
You need to run edit-config so you get your own copy in the user settings. See Configure the Netdata Agent | Learn Netdata
That’s the only way?
I put my custom netdata.conf in place by using Ansible because I have many many nodes at this point, and so orchestration (Ansible or SSH commands) is generally how I do everything.
I guess I would ask for a clear warning on that page, because, when I read that page prior to doing any of this, I figured I could interact/edit/replace /etc/netdata/netdata.conf to my hearts desire as I wasn’t editing the “stock config” as this warning tells you about:
Do not edit the files in /usr/lib/netdata/conf.d, as they are overwritten by updates to the Netdata Agent.
Which, I am not doing. I am editing /etc/netdata/netdata.conf, because I was trying to avoid this issue in the first place. I suppose that there just isn’t any way to distribute a changed configuration file to hundreds of servers at once. I’ll start logging into my machines one by one and resolving this.
Well, I went to my first out of date server and did this:
$ cd /etc/netdata
$ sudo ./edit-config netdata.conf
I verified my custom section expanding the dbengine was still there, and then closed the editor. When I closed & saved, it gave me this little prompt:
Editing '/etc/netdata/netdata.conf' ...
Then, I upgraded Netdata, and was immediately greeted by the same prompt:
Configuration file ‘/etc/netdata/netdata.conf’
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer’s version
N or O : keep your currently-installed version
Seemingly doing the same thing after using edit-config.
Actually, you’re right. The files directly under /etc/netdata should never be overwritten by an update. When executed automatically via the updater they must be maintained, when an interactive update happens it’s supposed to ask a question.
For Ansible, I see that there’s a documented way of doing configuration changes here: community/netdata-agent-deployment/ansible-quickstart at main · netdata/community · GitHub
What gets me a bit here is the apt update. The instructions for an ansible deployment say that kickstart should be used. Your version suggests that you didn’t use a package manager to install Netdata. So how does apt update do anything for netdata? @Austin_Hemmelgarn am I missing something here?
I’ve confused you, my apologies.
I’ve been using Netdata for a very long time and created my own installation method using the packagecloud repos before the kickstart script used them. Nothing about what I am doing is an “ansible deployment” as you have linked. The only reason I mentioned Ansible was so folks could grasp how I was interacting/replacing netdata.conf.
This is how I install Netdata onto debian/ubuntu, which is just a section from my master install script that prepares these machines to be deployed into the field.
curl -s https://packagecloud.io/install/repositories/netdata/netdata/script.deb.sh | bash
apt update
apt install netdata -y
netdata-claim.sh -token=value -rooms=value2 -url=value3
unzip -o /media/user/Ventoy/configs/netdconfigs-nvidia.zip -d /etc/netdata
systemctl enable --now netdata
Installing it in this way allows me to do all updates at a very specific time that I can control via “unattended-upgrades” as it hooks into the apt system.
I completely understand if your reaction is “Don’t do that”, I just never switched to using kickstart.sh because I didn’t want to read the entire kickstart.sh and understand exactly what it may be doing to my servers, but, I may just have to do that.
Regarding the configuration, it should be getting preserved properly for upgrades with native packages, but that’s dependent on how the tooling doing the upgrade handles things. In particular, some automation tooling for updates on Debian/Ubuntu systems may choose to install the version of config files from the package when prompted instead of preserving the user configuration.
Regarding the installation in general, if you want to avoid the kickstart script, we now provide packages with repository configuration (you can find them alongside the regular netdata packages, or alternatively in this dedicated repo). We’re in the process of setting up self-hosting of our package repositories, and using those packages instead of PackageCloud’s installation script will ensure you can migrate seamlessly without needing to do anything when we finally switch things over. These days, on any system we’re building native packages for, the kickstart script just fetches the appropriate repository configuration package, installs it, and then installs Netdata through the system PM (and optionally sets up automatic updates, which you can easily disable, and also optionally claims the node for you).