agent installed but "Command 'netdata' not found,"

Help
, ,
1

I was trying to reconfigure an agent to disable the web UI and couldn’t get the config changes to apply. I looked for the logs and they turned out to have moved to under:
/opt/netdata/etc/netdata

When I run the netdata command I get:

luis@pinode4:/opt/netdata/etc/netdata$ netdata
Command 'netdata' not found, but can be installed with:
sudo apt install netdata-core

but…the service is running (and web UI reachable):

luis@pinode4:/opt/netdata/etc/netdata$ sudo systemctl status netdata
● netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-04-30 22:27:15 BST; 2min 29s ago
    Process: 18166 ExecStartPre=/bin/mkdir -p /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 18167 ExecStartPre=/bin/chown -R netdata /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 18168 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
    Process: 18169 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
   Main PID: 18170 (netdata)
      Tasks: 86 (limit: 4368)
     Memory: 109.9M
        CPU: 41.733s
     CGroup: /system.slice/netdata.service

Once I modified the config at: /opt/netdata/etc/netdata/netdata.conf
then the changes took effect.
So what’s happened here?
The node is part of a k8s cluster which I’ve been testing netdata helm deployments to.
I’ve noticed that the helm files/charts use most of the same target directories as the native agent (which is probably a bad idea) and so I’m assuming that at some point I must have reinstalled the native agent and maybe it found the files/directories in use and fell-back to the /opt directory. The last bit is just a guess, of course.

I’d provide the build method but I can’t run the command :smiley:
OS: Ubuntu 22.10
Netdata: v1.38.0-450-g2d678553f

Interestingly, when I check the agent info via the API (http://:19999/api/v2/nodes) it indicates that:

"_is_k8s_node":"false",

I’m running k3s on the node and so I’m guessing that should say “true”?
Also, would it be worthwhile to add the build type info into the data pumped out by that API?

2

Hi, @Luis_Johnstone.

netdata is not in PATH when installed under /opt. This is expected.

I guess this is expected too, your Netdata is not part of k8s cluster. This parameter is true when Netdata is running inside K8s (in a pod).

3

Okay, here’s how it’s implemented:

  • env vars KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT are set. These two are always set in a Pod (in-cluster installation) OR
  • kubelet process is running on the node (out-of-cluster installation)
4

@ilyam8
Thanks for confirming that info. Good stuff :slight_smile:
I guess it still remains that now I have a slightly odd install.
How can I sort that out without losing any data?
Also, if I’m right about how it occurred then wouldn’t it make sense to change the helm charts to use different mount-points than the native agent? :smiley:
Am happy to do that and test it all works, if you are working on other stuff right now.

5

Can you elaborate a bit about your installation? How did you install it? Can you exec into netdata-child container and show env?

6

I re-installed via the kickstarter script:

wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --reinstall

netdata for k8s is currently not deployed (I’ve been installing and removing it for testing etc).

7

@ilyam8
Any suggestions about how I can get the agent back into a more normal state, preferably without losing data?

8

And what is the current state?

9

It’s currently a “kickstart-static” build type with a location for binaries outside of the usual path (as in a previous post above).
I didn’t choose that install type and, as above, I suspect it’s happened because of having netdata deployed in k8s at the same time as running a kickstarter reinstall on the host node.
So desired state would just be the default/vanilla for Install type: binpkg-deb.

10

Well, remove “kickstart-static” build and reinstall.

# remove old installation
sudo systemctl stop netdata
sudo rm -rf /opt/netdata
sudo rm /etc/cron.daily/netdata
sudo userdel -r netdata

# install new
...