Unable to start netdata service

stemy2 · November 26, 2020, 6:46pm

Hello,

I can’t load the monitors, the app is blocked on the splash screen. According my webmin, the app is marked as “auto-start”, and in journalctl, i got this:

Nov 26 16:55:45 stemy.me systemd[1]: netdata.service: Scheduled restart job, restart counter is at 23477.
Nov 26 16:55:45 stemy.me systemd[1]: Stopped Real time performance monitoring.
Nov 26 16:55:45 stemy.me systemd[1]: Starting Real time performance monitoring...
Nov 26 16:55:45 stemy.me systemd[1]: Started Real time performance monitoring.
Nov 26 16:55:45 stemy.me netdata[32482]: CONFIG: cannot load cloud config '/opt/netdata/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/opt/netdata/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Nov 26 16:55:45 stemy.me netdata[32482]: SIGNAL: Not enabling reaper
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : SIGNAL: Not enabling reaper
Nov 26 16:55:45 stemy.me netdata[32482]: LISTENER: IPv4 bind() on ip '0.0.0.0' port 19999, socktype 1 failed.
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata ERROR : MAIN : LISTENER: IPv4 bind() on ip '0.0.0.0' port 19999, socktype 1 failed. (errno 98, Address already in use)
Nov 26 16:55:45 stemy.me netdata[32482]: LISTENER: Cannot bind to ip '0.0.0.0', port 19999
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata ERROR : MAIN : LISTENER: Cannot bind to ip '0.0.0.0', port 19999
Nov 26 16:55:45 stemy.me netdata[32482]: LISTENER: IPv6 bind() on ip '::' port 19999, socktype 1 failed.
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata ERROR : MAIN : LISTENER: IPv6 bind() on ip '::' port 19999, socktype 1 failed. (errno 98, Address already in use)
Nov 26 16:55:45 stemy.me netdata[32482]: LISTENER: Cannot bind to ip '::', port 19999
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata ERROR : MAIN : LISTENER: Cannot bind to ip '::', port 19999
Nov 26 16:55:45 stemy.me netdata[32482]: LISTENER: Cannot listen on any API socket. Exiting...
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata FATAL : netdata : LISTENER: Cannot listen on any API socket. Exiting... # : Success
Nov 26 16:55:45 stemy.me netdata[32482]: EXIT: netdata prepares to exit with code 1...
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
Nov 26 16:55:45 stemy.me netdata[32482]: EXIT: cleaning up the database...
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : EXIT: cleaning up the database...
Nov 26 16:55:45 stemy.me netdata[32482]: Cleaning up database [0 hosts(s)]...
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
Nov 26 16:55:45 stemy.me netdata[32482]: EXIT: removing netdata PID file '/opt/netdata/var/run/netdata/netdata.pid'...
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : EXIT: removing netdata PID file '/opt/netdata/var/run/netdata/netdata.pid'...
Nov 26 16:55:45 stemy.me netdata[32482]: EXIT: cannot unlink pidfile '/opt/netdata/var/run/netdata/netdata.pid'.
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata ERROR : MAIN : EXIT: cannot unlink pidfile '/opt/netdata/var/run/netdata/netdata.pid'. (errno 2, No such file or directory)
Nov 26 16:55:45 stemy.me netdata[32482]: EXIT: all done - netdata is now exiting - bye bye...
Nov 26 16:55:45 stemy.me netdata[32482]: 2020-11-26 16:55:45: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...
Nov 26 16:55:45 stemy.me systemd[1]: netdata.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 16:55:45 stemy.me systemd[1]: netdata.service: Failed with result 'exit-code'.

OS / Environment

Linux stemy.me 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux
etc/os-release:PRETTY_NAME=“Debian GNU/Linux 10 (buster)”
(actually Yunohost, wich is debian based)

Netdata version

1.26

Thiago_Marques_0 · November 26, 2020, 7:18pm

Hello @stemy2,

According to these messages, it looks like that some process is already running and listening the port 19999, so Netdata cannot start using it. Please, can you run the command netstat -antp and give us the output? This will help us to identify the software that is listening the port.

Best regards!

stemy2 · November 26, 2020, 7:30pm

The command netstat is not found and is not installable from my repositories.

Thiago_Marques_0 · November 26, 2020, 10:43pm

This does the things a little bit complicate, but let us move forward. I won’t require you to install any new package, please, run the following commands:

$ ps aux| grep netdata
$ sudo systemctl stop netdata
$ cat /proc/net/tcp | grep 4E1F

The first command will display if there is netdata process running.
The second command will stop any Netdata running, if the first command showed that there was a process running.
Finally we will check if there is a different process listen top the port 19999( Hex value 4E1F).
If the last step does not show any port, you will be able to start Netdata.

These tests are assuming that Netdata is up and running, so we are trying to open an instance before to start a new.

Best regards!

stemy2 · November 26, 2020, 10:57pm

OK, there is the command output:

root@stemy:~# cat /proc/net/tcp | grep 4E1F
  35: 00000000:4E1F 00000000:0000 0A 00000000:00000011 00:00000000 00000000     0        0 36449622 18 000000009a45c2f4 100 0 0 10 0                 
  81: 0100007F:4E1F 0100007F:C11A 08 00000000:000001B2 00:00000000 00000000     0        0 0 1 00000000f4166c87 20 4 30 10 9                         
 111: 0100007F:4E1F 0100007F:C120 08 00000000:00000193 00:00000000 00000000     0        0 0 1 000000001ee903f9 20 4 30 10 9                         
 112: 0100007F:4E1F 0100007F:D15C 08 00000000:000001D3 00:00000000 00000000     0        0 0 1 00000000d2a11cb7 20 4 30 10 9                         
 116: 0100007F:4E1F 0100007F:D3A6 08 00000000:00000001 00:00000000 00000000     0        0 0 1 0000000031a91d22 20 4 0 10 9                          
 144: 0100007F:4E1F 0100007F:D14E 08 00000000:000001BB 00:00000000 00000000     0        0 0 1 00000000f43f0ae1 20 4 30 10 9                         
 166: 0100007F:C11A 0100007F:4E1F 05 00000000:00000000 03:00000926 00000000     0        0 0 3 0000000003dd260e                                      
 173: 0100007F:4E1F 0100007F:C122 08 00000000:000001BB 00:00000000 00000000     0        0 0 1 00000000fa7362c0 20 4 30 10 9                         
 176: 0100007F:4E1F 0100007F:C11E 08 00000000:00000001 00:00000000 00000000     0        0 0 1 00000000db5d37ef 20 4 0 10 9                          
 186: 0100007F:C122 0100007F:4E1F 05 00000000:00000000 03:00000926 00000000     0        0 0 3 000000003f598ec5                                      
 193: 0100007F:4E1F 0100007F:D3B4 08 00000000:000001BB 00:00000000 00000000     0        0 0 1 00000000fce53222 20 4 30 10 9                         
 220: 0100007F:4E1F 0100007F:C116 08 00000000:00000001 00:00000000 00000000     0        0 0 1 00000000b8144618 20 4 0 10 9                          
 231: 0100007F:4E1F 0100007F:D3AA 08 00000000:00000001 00:00000000 00000000     0        0 0 1 00000000635ded36 20 4 0 10 9                          
 234: 0100007F:4E1F 0100007F:D3B0 08 00000000:000001B4 00:00000000 00000000     0        0 0 1 00000000a3f120b4 20 4 30 10 9                         
 236: 0100007F:4E1F 0100007F:C118 08 00000000:000001C3 00:00000000 00000000     0        0 0 1 0000000071c75d40 20 4 30 10 9                         
 243: 0100007F:C118 0100007F:4E1F 05 00000000:00000000 03:00000926 00000000     0        0 0 3 000000003a2ae639                                      
 248: 0100007F:4E1F 0100007F:C114 08 00000000:00000001 00:00000000 00000000     0        0 0 1 00000000b76b5a81 20 4 0 10 9                          
 263: 0100007F:C114 0100007F:4E1F 05 00000000:00000000 03:00000801 00000000     0        0 0 3 000000004f24801e                                      
 268: 0100007F:4E1F 0100007F:D3AE 08 00000000:000001B2 00:00000000 00000000     0        0 0 1 000000006d7a2c4e 20 4 30 10 9                         
 275: 0100007F:4E1F 0100007F:D3AC 08 00000000:000001C3 00:00000000 00000000     0        0 0 1 0000000025062bf8 20 4 30 10 9                         
 282: 0100007F:C120 0100007F:4E1F 05 00000000:00000000 03:00000926 00000000     0        0 0 3 00000000bd0a2229                                      
 285: 0100007F:C11E 0100007F:4E1F 05 00000000:00000000 03:0000080E 00000000     0        0 0 3 00000000fb9627ed                                      
 298: 0100007F:C11C 0100007F:4E1F 05 00000000:00000000 03:00000926 00000000     0        0 0 3 00000000487af273                                      
 309: 0100007F:4E1F 0100007F:C11C 08 00000000:000001B4 00:00000000 00000000     0        0 0 1 00000000b5c5bc80 20 4 30 10 9                         
 312: 0100007F:C116 0100007F:4E1F 05 00000000:00000000 03:00000801 00000000     0        0 0 3 000000000777c349                                      
 314: 0100007F:4E1F 0100007F:D3B2 08 00000000:00000193 00:00000000 00000000     0        0 0 1 000000001521b8c9 20 4 30 10 9

Thiago_Marques_0 · November 27, 2020, 8:55pm

According this output, we have 18 incoming connections for Netdata, probably this is the motive you cannot access it. Are you using a script or other software to call Netdata dashboard?

This last output also did not help us to discover if there was a Netdata running, or another process using this port. Did you run the other two commands?

Finally, do you have other Netdata processes streaming for this host?

Best regards!

stemy2 · November 27, 2020, 9:07pm

I honestly can’t answer to your question. Netdata worked fine until the problem appeared out of the blue.

Thiago_Marques_0 · December 2, 2020, 1:05pm

Hello @stemy2,

Sorry for the delay, but I am at vacation, let me try to help you.

It looks like that Netdata is running, but the sockets opened somehow cannot be closed and it is not possible to open another connection. When you run the following commands:

$ sudo systemctl stop netdata
$ ps aux| grep netdata

after the second command, can you see a netdata process running? If there are, we are not able to stop completely Netdata, please, run the next command:

$ sudo killall netdata
$ ps aux| grep netdata

It is expected that after the next command sequence, Netdata won’t be running, after this, try to start Netdata again:

$ sudo systemctl start netdata

Please, let us know if this helped you.

I am in vacation next few days, but I will try to answer you. I am really sorry for this delay to answer you!

Best regards!

Topic		Replies	Views
Netdata not picking up one service Help agent	2	566	January 11, 2021
Issue in netdata as container in cloud Help container-monitoring , agent	8	3684	September 14, 2020
Current issues under Rocky 9 and docker Help agent , collectors	15	1802	January 22, 2023
Netdata Container Falies to Start Help agent	3	672	June 9, 2021
Netdata Cloud - VMWare vCenter not logging in platform after "successful" installation Help agent , cloud , configuration , installation	9	1038	October 3, 2022

Unable to start netdata service

OS / Environment

Netdata version

Related Topics