Bare Metal Server with netdata Agent installed netdata v1.40.0-70-nightly
===
I have a problem with some charts.
I am sure that until at least July 18 the following charts were there:
apache_local.workers
I can’t find anything in the logs of netdata.
In the current netdata I find under the heading web log apache and web log apache vhost of course some charts, but earlier there was a separate section for the workers of Apache.
With I think 2 dimensions idle and busy .
Does apache_local.workers still exist?
How can I get that again (multiple restart of netdata is already done)
and /etc/netdata/go.d/apache.conf has the parameter " autodetection_retry: 120" with 1 space at the beginning.
If in the newer versions the chart is not there anymore, where and how do I get something like this?
I did not notice it in the changelog. May be I missed that.
`autodetection_retry`
space removed, Thought somewhere I found the space was mandatory . My failure
using debug mode again.
https://paste.debian.net/hidden/87cd6b1f/ first 2000 lines
Thank you,
yes my Apache workers are back!
Of course I could have found out about the space myself. However, it takes some time until it is visible as a result.
With the next problem I will search more carefully, I promise.
First my congratulations to the successful local surface!
Sorry to bother again.
Today I have with the
netdata v1.40.0-156-nightly
again the problem that the apache charts are not displayed. The debug looks like values are delivered…
However, they do not reach the charts. See screenshot. Nothing was changed in the config /etc/netdata/go.d/apache.conf.
$ ./go.d.plugin -d -m apache
[ DEBUG ] main[main] main.go:115 plugin: name=go.d, version=v0.54.0
[ DEBUG ] main[main] main.go:117 current user: name=netdata, uid=115
[ INFO ] main[main] main.go:121 env HTTP_PROXY '', HTTPS_PROXY '' (both upper and lower case are respected)
[ INFO ] main[main] agent.go:140 instance is started
[ INFO ] main[main] setup.go:43 loading config file
[ INFO ] main[main] setup.go:51 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO ] main[main] setup.go:58 found '/etc/netdata/go.d.conf
[ INFO ] main[main] setup.go:65 config successfully loaded
[ INFO ] main[main] agent.go:144 using config: enabled 'true', default_run 'true', max_procs '0'
[ INFO ] main[main] setup.go:70 loading modules
[ INFO ] main[main] setup.go:89 enabled/registered modules: 1/79
[ INFO ] main[main] setup.go:94 building discovery config
[ INFO ] main[main] setup.go:141 looking for 'apache.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[ INFO ] main[main] setup.go:157 found '/etc/netdata/go.d/apache.conf
[ INFO ] main[main] setup.go:162 dummy/read/watch paths: 0/1/0
[ INFO ] discovery[manager] manager.go:92 registered discoverers: [file discovery: [file reader]]
[ INFO ] main[main] setup.go:176 looking for 'vnodes/' in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO ] main[main] setup.go:188 found '/usr/lib/netdata/conf.d/vnodes' (0 vhosts)
[ INFO ] discovery[manager] manager.go:97 instance is started
[ INFO ] run[manager] run.go:32 instance is started
[ INFO ] build[manager] build.go:117 instance is started
[ INFO ] discovery[file manager] discovery.go:73 instance is started
[ INFO ] discovery[file reader] read.go:41 instance is started
[ INFO ] discovery[file reader] read.go:42 instance is stopped
[ DEBUG ] build[manager] build.go:164 received config group ('/etc/netdata/go.d/apache.conf'): 2 jobs (added: 2, removed: 0)
[ DEBUG ] build[manager] build.go:313 building apache[local] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/apache.conf autodetection_retry:120 module:apache name:local priority:70000 update_every:1 url:http://localhost/server-status?auto]
[ DEBUG ] apache[local] apache.go:65 using URL http://localhost/server-status?auto
[ DEBUG ] apache[local] apache.go:66 using timeout: 2s
[ INFO ] apache[local] job.go:208 check success
[ INFO ] build[manager] build.go:219 apache[local] job is being served by another job, skipping it
[ INFO ] apache[local] job.go:228 started, data collection interval 1s
[ DEBUG ] run[manager] run.go:43 tick 0
CHART 'netdata.execution_time_of_apache_local' '' 'Execution time' 'ms' 'go.d' 'netdata.go_plugin_execution_time' 'line' '145000' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'time' '' 'absolute' '1' '1' ''
CHART 'apache_local.workers' '' 'Workers Threads' 'workers' 'workers' 'apache.workers' 'stacked' '70005' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'idle' 'idle' 'absolute' '1' '1' ''
DIMENSION 'busy' 'busy' 'absolute' '1' '1' ''
BEGIN 'apache_local.workers'
SET 'idle' = 6
SET 'busy' = 6
END
CHART 'apache_local.scoreboard' '' 'Scoreboard' 'connections' 'connections' 'apache.scoreboard' 'line' '70003' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'waiting' 'waiting' 'absolute' '1' '1' ''
DIMENSION 'starting' 'starting' 'absolute' '1' '1' ''
DIMENSION 'reading' 'reading' 'absolute' '1' '1' ''
DIMENSION 'sending' 'sending' 'absolute' '1' '1' ''
DIMENSION 'keepalive' 'keepalive' 'absolute' '1' '1' ''
DIMENSION 'dns_lookup' 'dns_lookup' 'absolute' '1' '1' ''
DIMENSION 'closing' 'closing' 'absolute' '1' '1' ''
DIMENSION 'logging' 'logging' 'absolute' '1' '1' ''
DIMENSION 'finishing' 'finishing' 'absolute' '1' '1' ''
DIMENSION 'idle_cleanup' 'idle_cleanup' 'absolute' '1' '1' ''
DIMENSION 'open' 'open' 'absolute' '1' '1' ''
BEGIN 'apache_local.scoreboard'
SET 'waiting' = 6
SET 'starting' = 0
SET 'reading' = 1
SET 'sending' = 1
SET 'keepalive' = 3
SET 'dns_lookup' = 0
SET 'closing' = 0
SET 'logging' = 0
SET 'finishing' = 1
SET 'idle_cleanup' = 0
SET 'open' = 138
END
CHART 'apache_local.requests' '' 'Requests' 'requests/s' 'requests' 'apache.requests' 'line' '70000' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'requests' 'requests' 'incremental' '1' '1' ''
BEGIN 'apache_local.requests'
SET 'requests' = 753922
END
CHART 'apache_local.net' '' 'Bandwidth' 'kilobits/s' 'bandwidth' 'apache.net' 'area' '70004' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'sent' 'sent' 'incremental' '8' '1' ''
BEGIN 'apache_local.net'
SET 'sent' = 45023481
END
CHART 'apache_local.reqpersec' '' 'Lifetime Average Number Of Requests Per Second' 'requests/s' 'statistics' 'apache.reqpersec' 'area' '70006' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'requests' 'requests' 'absolute' '1' '100000' ''
BEGIN 'apache_local.reqpersec'
SET 'requests' = 246138
END
CHART 'apache_local.bytespersec' '' 'Lifetime Average Number Of Bytes Served Per Second' 'KiB/s' 'statistics' 'apache.bytespersec' 'area' '70007' '1' '' 'go.d' 'apache'
CLABEL '_collect_job' 'local' '1'
CLABEL_COMMIT
DIMENSION 'served' 'served' 'absolute' '8' '102400000' ''
I have attached the new screenshot of 19999. The time of the disappearance is about 4:30. In the period in question I find only this in collector.log.1
2023-07-13 03:04:31: charts.d: ERROR: opensips: Die Funktion 'opensips' check() des Moduls meldet einen Fehler.
2023-07-13 06:31:00: netdata INFO : P[idlejitter] : Aufräumen...
In collector.2.gz
2023-07-13 02:00:45: apps.plugin ERROR : apps.plugin : Kann /proc/217495/cmdline (Befehl 'rkhunter') nicht verarbeiten (errno 3, No such process)
2023-07-13 04:24:38: go.d INFO: web_log[apache_vhosts] unmatched line: csv parse: parse error on line 216729, column 232: extraneous or missing " in quoted-field (parser: csv: $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time)
2023-07-13 04:24:39: go.d INFO: web_log[apache_vhosts] unmatched line: csv parse: parse error on line 216731, column 206: extraneous or missing " in quoted-field (parser: csv: $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time)
This does not seem very suspicious to me
the screenshot:
I’ll put it this way: My fingers were itching for a restart before I complained. But I decided against it, because the nightly update was up to date.
That by the way. I have now made the restart, waited a bit, reload the browser and behold, the charts are filled again.
But maybe you still have an idea, where you can recognize that a restart is necessary.
In any case, thanks for your expertise!
Hello @ilyam8:
I looked at it again, it seems that when apache again shows no charts, it happens after an upgrade and the corresponding restart. If I then manually restart the world is fine again. Until there is again an upgrade.
It is otherwise somehow related to this,
when I look at syslog, I see the restart always around 6:45. Is it possible that the restart after an upgrade is different from the manual restart I am sorry, the title was and is misleading.
It concerns not only apache, but quite 50% other charts.
E.G."
system.processes_state- [processes]
Apps CPU time (100% = 1 core)- apps.cpu- [percent]
Apps CPU user time (100% = 1 core)- apps.cpu_user- [percent]
pps voluntary context switches- apps.voluntary_ctxt_switches- [k switches/s]
Apps disk reads- apps.preads- [MiB/s]
and much more until
Containers & VMs are displayed
fail2ban is displayed
and everything else up to apache
dns query is not shown
docker container is not shown
mysql and postgres are not shown either
webserver_logs are not shown either
Until netdata itself, these will be there again.
"
Hope this helps you guys with the analysis.
My workaround is a restart. Why this occurs only after the netdata upgrade? No idea
