Apache configuration - no work threads information available!?

Cedric_Counotte · May 23, 2022, 11:33am

Hello all,

I am monitoring 5 servers with netdata and I’m very happy about the whole solution. Though I have an issue with apache monitoring which I cannot explain nor fix.

On 3 servers (out of 5) I can select apache_local.workers to monitor active workers, but on 2 of them, the data is just not there!?

This page explains how to configure apache plugin, but I can’t find the apache.conf file on any servers:
Apache monitoring with Netdata | Learn Netdata

I can load http://127.0.0.1/server-status?auto on any of the 5 servers so getting the data should be no problem.

On the 2 servers without apache workers data the following data are listed , if that can help:
apache_local.connections
apache_local.conns_async

All 5 servers have the same version of Apache, same version of netdata and are supposedly configured identically (from a common install script).

Any help in solving this will be greatly appreciated

Tasos_Katsoulas · May 23, 2022, 11:39am

Hi there,

Could you please run this Apache monitoring with Netdata | Learn Netdata troubleshooting section in two of your nodes that present different behavior (one that it monitors the active workers and one that it doesn’t) and send us the logs (Hide any sensitive information, public IPs etc)? Could you also provide me with the netdata -W buildinfo for those nodes?

Tasos.

Cedric_Counotte · May 23, 2022, 12:05pm

Thanks for your quick reply, here are the requested information.

Strangely the data appears to be collected, however I can assure you when I go to dashboard Netdata Console… it’s not there!? I don’t use local pages at all so I can’t tell if the local monitoring page would be showing it.

One one of the 2 servers I’m (now?) getting scoreboard and uptime, but still no worker thread.

Version: netdata v1.34.0-212-nightly
Configure options: ‘–build=x86_64-linux-gnu’ ‘–includedir=${prefix}/include’ ‘–mandir=${prefix}/share/man’ ‘–infodir=${prefix}/share/info’ ‘–disable-option-checking’ ‘–disable-silent-rules’ ‘–libdir=${prefix}/lib/x86_64-linux-gnu’ ‘–libexecdir=${prefix}/lib/x86_64-linux-gnu’ ‘–disable-maintainer-mode’ ‘–prefix=/usr’ ‘–sysconfdir=/etc’ ‘–localstatedir=/var’ ‘–libdir=/usr/lib’ ‘–libexecdir=/usr/libexec’ ‘–with-user=netdata’ ‘–with-math’ ‘–with-zlib’ ‘–with-webdir=/var/lib/netdata/www’ ‘–disable-dependency-tracking’ ‘build_alias=x86_64-linux-gnu’ ‘CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’ ‘LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro’ ‘CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2’ ‘CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’
Install type: binpkg-deb
Binary architecture: x86_64
Packaging distro:
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK Next Generation: YES
ACLK-NG New Cloud Protocol: YES
ACLK Legacy: NO
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: YES
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: YES
IPMI: YES
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES

[ DEBUG ] main[main] main.go:111 plugin: name=go.d, version=v0.32.3
[ DEBUG ] main[main] main.go:113 current user: name=netdata, uid=118
[ INFO ] main[main] agent.go:106 instance is started
[ INFO ] main[main] setup.go:40 loading config file
[ INFO ] main[main] setup.go:48 looking for ‘go.d.conf’ in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO ] main[main] setup.go:55 found ‘/usr/lib/netdata/conf.d/go.d.conf
[ INFO ] main[main] setup.go:62 config successfully loaded
[ INFO ] main[main] agent.go:110 using config: enabled ‘true’, default_run ‘true’, max_procs ‘0’
[ INFO ] main[main] setup.go:67 loading modules
[ INFO ] main[main] setup.go:86 enabled/registered modules: 1/65
[ INFO ] main[main] setup.go:91 building discovery config
[ INFO ] main[main] setup.go:121 looking for ‘apache.conf’ in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[ INFO ] main[main] setup.go:137 found ‘/usr/lib/netdata/conf.d/go.d/apache.conf
[ INFO ] main[main] setup.go:142 dummy/read/watch paths: 0/1/0
[ INFO ] discovery[manager] manager.go:90 registered discoverers: [file discovery: [file reader]]
[ INFO ] discovery[manager] manager.go:95 instance is started
[ INFO ] run[manager] run.go:30 instance is started
[ INFO ] build[manager] build.go:106 instance is started
[ INFO ] discovery[file manager] discovery.go:71 instance is started
[ INFO ] discovery[file reader] read.go:39 instance is started
[ INFO ] discovery[file reader] read.go:40 instance is stopped
[ DEBUG ] build[manager] build.go:153 received config group (’/usr/lib/netdata/conf.d/go.d/apache.conf’): 2 jobs (added: 2, removed: 0)
[ DEBUG ] build[manager] build.go:302 building apache[local] job, config: map[provider:file reader source:/usr/lib/netdata/conf.d/go.d/apache.conf autodetection_retry:0 module:apache name:local priority:70000 update_every:1 url:http://localhost/server-status?auto]
[ DEBUG ] apache[local] apache.go:79 using URL http://localhost/server-status?auto
[ DEBUG ] apache[local] apache.go:80 using timeout: 2s
[ INFO ] apache[local] job.go:171 check success
[ INFO ] build[manager] build.go:208 apache[local] job is being served by another job, skipping it
[ INFO ] apache[local] job.go:191 started, data collection interval 1s
[ DEBUG ] run[manager] run.go:41 tick 0
CHART ‘netdata.execution_time_of_apache_local’ ‘’ ‘Execution time’ ‘ms’ ‘go.d’ ‘netdata.go_plugin_execution_time’ ‘’ ‘145000’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘time’ ‘’ ‘’ ‘1’ ‘1’ ‘’

CHART ‘apache_local.requests’ ‘’ ‘Requests’ ‘requests/s’ ‘requests’ ‘apache.requests’ ‘’ ‘70000’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘total_accesses’ ‘requests’ ‘incremental’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.requests’
SET ‘total_accesses’ = 35520740
END

CHART ‘apache_local.connections’ ‘’ ‘Connections’ ‘connections’ ‘connections’ ‘apache.connections’ ‘’ ‘70001’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘conns_total’ ‘connections’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.connections’
SET ‘conns_total’ =
END

CHART ‘apache_local.conns_async’ ‘’ ‘Async Connections’ ‘connections’ ‘connections’ ‘apache.conns_async’ ‘stacked’ ‘70002’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘conns_async_keep_alive’ ‘keepalive’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘conns_async_closing’ ‘closing’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘conns_async_writing’ ‘writing’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.conns_async’
SET ‘conns_async_keep_alive’ =
SET ‘conns_async_closing’ =
SET ‘conns_async_writing’ =
END

CHART ‘apache_local.scoreboard’ ‘’ ‘Scoreboard’ ‘connections’ ‘connections’ ‘apache.scoreboard’ ‘’ ‘70003’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘scoreboard_waiting’ ‘waiting’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_starting’ ‘starting’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_reading’ ‘reading’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_sending’ ‘sending’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_keepalive’ ‘keepalive’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_dns_lookup’ ‘dns lookup’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_closing’ ‘closing’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_logging’ ‘logging’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_finishing’ ‘finishing’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_idle_cleanup’ ‘idle cleanup’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_open’ ‘open’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.scoreboard’
SET ‘scoreboard_waiting’ = 6
SET ‘scoreboard_starting’ = 0
SET ‘scoreboard_reading’ = 3
SET ‘scoreboard_sending’ = 2
SET ‘scoreboard_keepalive’ = 56
SET ‘scoreboard_dns_lookup’ = 0
SET ‘scoreboard_closing’ = 6
SET ‘scoreboard_logging’ = 0
SET ‘scoreboard_finishing’ = 0
SET ‘scoreboard_idle_cleanup’ = 0
SET ‘scoreboard_open’ = 77
END

CHART ‘apache_local.net’ ‘’ ‘Bandwidth’ ‘kilobits/s’ ‘bandwidth’ ‘apache.net’ ‘area’ ‘70004’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘total_kBytes’ ‘sent’ ‘incremental’ ‘8’ ‘1’ ‘’

BEGIN ‘apache_local.net’
SET ‘total_kBytes’ = 183949352
END

CHART ‘apache_local.workers’ ‘’ ‘Workers Threads’ ‘workers’ ‘workers’ ‘apache.workers’ ‘stacked’ ‘70005’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘idle_workers’ ‘idle’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘busy_workers’ ‘busy’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.workers’
SET ‘idle_workers’ = 6
SET ‘busy_workers’ = 67
END

CHART ‘apache_local.reqpersec’ ‘’ ‘Lifetime Average Number Of Requests Per Second’ ‘requests/s’ ‘statistics’ ‘apache.reqpersec’ ‘area’ ‘70006’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘req_per_sec’ ‘requests’ ‘’ ‘1’ ‘100000’ ‘’

BEGIN ‘apache_local.reqpersec’
SET ‘req_per_sec’ = 2301890
END

CHART ‘apache_local.bytespersec’ ‘’ ‘Lifetime Average Number Of Bytes Served Per Second’ ‘KiB/s’ ‘statistics’ ‘apache.bytespersec’ ‘area’ ‘70007’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘bytes_per_sec’ ‘served’ ‘’ ‘8’ ‘102400000’ ‘’

BEGIN ‘apache_local.bytespersec’
SET ‘bytes_per_sec’ = 12206700000
END

CHART ‘apache_local.bytesperreq’ ‘’ ‘Lifetime Average Response Size’ ‘KiB’ ‘statistics’ ‘apache.bytesperreq’ ‘area’ ‘70008’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘bytes_per_req’ ‘size’ ‘’ ‘1’ ‘102400000’ ‘’

BEGIN ‘apache_local.bytesperreq’
SET ‘bytes_per_req’ = 530293000
END

CHART ‘apache_local.uptime’ ‘’ ‘Uptime’ ‘seconds’ ‘availability’ ‘apache.uptime’ ‘’ ‘70009’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘uptime’ ‘’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.uptime’
SET ‘uptime’ = 1543115
END

BEGIN ‘netdata.execution_time_of_apache_local’
SET ‘time’ = 0
END

[ DEBUG ] run[manager] run.go:41 tick 1
BEGIN ‘apache_local.requests’ 1000855
SET ‘total_accesses’ = 35520786
END

BEGIN ‘apache_local.connections’
SET ‘conns_total’ =
END

BEGIN ‘apache_local.conns_async’
SET ‘conns_async_keep_alive’ =
SET ‘conns_async_closing’ =
SET ‘conns_async_writing’ =
END

BEGIN ‘apache_local.scoreboard’ 1000855
SET ‘scoreboard_waiting’ = 7
SET ‘scoreboard_starting’ = 0
SET ‘scoreboard_reading’ = 1
SET ‘scoreboard_sending’ = 1
SET ‘scoreboard_keepalive’ = 53
SET ‘scoreboard_dns_lookup’ = 0
SET ‘scoreboard_closing’ = 11
SET ‘scoreboard_logging’ = 0
SET ‘scoreboard_finishing’ = 0
SET ‘scoreboard_idle_cleanup’ = 0
SET ‘scoreboard_open’ = 77
END

BEGIN ‘apache_local.net’ 1000855
SET ‘total_kBytes’ = 183949489
END

BEGIN ‘apache_local.workers’ 1000855
SET ‘idle_workers’ = 7
SET ‘busy_workers’ = 66
END

BEGIN ‘apache_local.reqpersec’ 1000855
SET ‘req_per_sec’ = 2301890
END

BEGIN ‘apache_local.bytespersec’ 1000855
SET ‘bytes_per_sec’ = 12206700000
END

BEGIN ‘apache_local.bytesperreq’ 1000855
SET ‘bytes_per_req’ = 530293000
END

BEGIN ‘apache_local.uptime’ 1000855
SET ‘uptime’ = 1543116
END

BEGIN ‘netdata.execution_time_of_apache_local’ 1000855
SET ‘time’ = 1
END

Tasos_Katsoulas · May 23, 2022, 12:25pm

I don’t use local pages at all so I can’t tell if the local monitoring page would be showing it.

I am not sure what steps you followed to restrict access to the dashboard, could you verify that we can see the charts in question under <node_ip>:19999/api/v1/charts?

If yes then this is a Netdata cloud issue.

Cedric_Counotte · May 23, 2022, 12:38pm

I did this: curl localhost:19999/api/v1/charts | grep apache_local.workers on both a working and non-working node and could see similar output from the two, here is the complete entry from the “non-working” node:

            "apache_local.workers":                 {
                    "id": "apache_local.workers",
                    "name": "apache_local.workers",
                    "type": "apache_local",
                    "family": "workers",
                    "context": "apache.workers",
                    "title": "Workers Threads (apache_local.workers)",
                    "priority": 70005,
                    "plugin": "go.d",
                    "module": "apache",
                    "units": "workers",
                    "data_url": "/api/v1/data?chart=apache_local.workers",
                    "chart_type": "stacked",
                    "duration": 1545094,
                    "first_entry": 1651764164,
                    "last_entry": 1653309257,
                    "update_every": 1,
                    "dimensions": {
                            "idle_workers": { "name": "idle" },
                            "busy_workers": { "name": "busy" }
                    },
                    "chart_variables": {
                    },
                    "green": null,
                    "red": null,
                    "alarms": {

                    },
                    "chart_labels": {

                    }

            },

Tasos_Katsoulas · May 23, 2022, 12:48pm

Opened an upstream issue [Bug]: Missing charts on Netdata Cloud · Issue #437 · netdata/netdata-cloud · GitHub you can follow the progress there.

Cedric_Counotte · May 23, 2022, 12:48pm

While investigating I noticed there is a similar issue with apache_local.scoreboard, 3 nodes are showing it out of 5, but it’s not the same nodes!
workers are shown on nodes 1, 2 and 5 and it’s listed in all servers v1/charts URL.
scoreboard is shown on nodes 1, 4 and 5 and it’s listed in all servers but 2! ?

Really odd.

PS: Thanks for opening the issue.

Topic		Replies	Views
missing apache_local.workers Help agent	12	320	July 21, 2023
Apache Plugin not showing up ? Help	2	26	April 12, 2024
Apache scoreboard gone Help	1	156	December 25, 2023
Missing statistics like 'users' Help agent , cloud , collectors	2	351	June 6, 2023
Web Server Logs and Systemd logs not available Help linux , platform	8	430	November 9, 2023

Apache configuration - no work threads information available!?

Related Topics