Apache configuration - no work threads information available!?

Help
,
1

Hello all,

I am monitoring 5 servers with netdata and I’m very happy about the whole solution. Though I have an issue with apache monitoring which I cannot explain nor fix.

On 3 servers (out of 5) I can select apache_local.workers to monitor active workers, but on 2 of them, the data is just not there!?

This page explains how to configure apache plugin, but I can’t find the apache.conf file on any servers:
Apache monitoring with Netdata | Learn Netdata

I can load http://127.0.0.1/server-status?auto on any of the 5 servers so getting the data should be no problem.

On the 2 servers without apache workers data the following data are listed , if that can help:
apache_local.connections
apache_local.conns_async

All 5 servers have the same version of Apache, same version of netdata and are supposedly configured identically (from a common install script).

Any help in solving this will be greatly appreciated :wink:

2

Hi there,

Could you please run this Apache monitoring with Netdata | Learn Netdata troubleshooting section in two of your nodes that present different behavior (one that it monitors the active workers and one that it doesn’t) and send us the logs (Hide any sensitive information, public IPs etc)? Could you also provide me with the netdata -W buildinfo for those nodes?

Tasos.

3

Thanks for your quick reply, here are the requested information.

Strangely the data appears to be collected, however I can assure you when I go to dashboard Netdata Console… it’s not there!? I don’t use local pages at all so I can’t tell if the local monitoring page would be showing it.

One one of the 2 servers I’m (now?) getting scoreboard and uptime, but still no worker thread.

Version: netdata v1.34.0-212-nightly
Configure options: ‘–build=x86_64-linux-gnu’ ‘–includedir=${prefix}/include’ ‘–mandir=${prefix}/share/man’ ‘–infodir=${prefix}/share/info’ ‘–disable-option-checking’ ‘–disable-silent-rules’ ‘–libdir=${prefix}/lib/x86_64-linux-gnu’ ‘–libexecdir=${prefix}/lib/x86_64-linux-gnu’ ‘–disable-maintainer-mode’ ‘–prefix=/usr’ ‘–sysconfdir=/etc’ ‘–localstatedir=/var’ ‘–libdir=/usr/lib’ ‘–libexecdir=/usr/libexec’ ‘–with-user=netdata’ ‘–with-math’ ‘–with-zlib’ ‘–with-webdir=/var/lib/netdata/www’ ‘–disable-dependency-tracking’ ‘build_alias=x86_64-linux-gnu’ ‘CFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’ ‘LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro’ ‘CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2’ ‘CXXFLAGS=-g -O2 -ffile-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security’
Install type: binpkg-deb
Binary architecture: x86_64
Packaging distro:
Features:
dbengine: YES
Native HTTPS: YES
Netdata Cloud: YES
ACLK Next Generation: YES
ACLK-NG New Cloud Protocol: YES
ACLK Legacy: NO
TLS Host Verification: YES
Machine Learning: YES
Stream Compression: YES
Libraries:
protobuf: YES (system)
jemalloc: NO
JSON-C: YES
libcap: NO
libcrypto: YES
libm: YES
tcalloc: NO
zlib: YES
Plugins:
apps: YES
cgroup Network Tracking: YES
CUPS: YES
EBPF: YES
IPMI: YES
NFACCT: YES
perf: YES
slabinfo: YES
Xen: NO
Xen VBD Error Tracking: NO
Exporters:
AWS Kinesis: NO
GCP PubSub: NO
MongoDB: NO
Prometheus Remote Write: YES

[ DEBUG ] main[main] main.go:111 plugin: name=go.d, version=v0.32.3
[ DEBUG ] main[main] main.go:113 current user: name=netdata, uid=118
[ INFO ] main[main] agent.go:106 instance is started
[ INFO ] main[main] setup.go:40 loading config file
[ INFO ] main[main] setup.go:48 looking for ‘go.d.conf’ in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO ] main[main] setup.go:55 found ‘/usr/lib/netdata/conf.d/go.d.conf
[ INFO ] main[main] setup.go:62 config successfully loaded
[ INFO ] main[main] agent.go:110 using config: enabled ‘true’, default_run ‘true’, max_procs ‘0’
[ INFO ] main[main] setup.go:67 loading modules
[ INFO ] main[main] setup.go:86 enabled/registered modules: 1/65
[ INFO ] main[main] setup.go:91 building discovery config
[ INFO ] main[main] setup.go:121 looking for ‘apache.conf’ in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[ INFO ] main[main] setup.go:137 found ‘/usr/lib/netdata/conf.d/go.d/apache.conf
[ INFO ] main[main] setup.go:142 dummy/read/watch paths: 0/1/0
[ INFO ] discovery[manager] manager.go:90 registered discoverers: [file discovery: [file reader]]
[ INFO ] discovery[manager] manager.go:95 instance is started
[ INFO ] run[manager] run.go:30 instance is started
[ INFO ] build[manager] build.go:106 instance is started
[ INFO ] discovery[file manager] discovery.go:71 instance is started
[ INFO ] discovery[file reader] read.go:39 instance is started
[ INFO ] discovery[file reader] read.go:40 instance is stopped
[ DEBUG ] build[manager] build.go:153 received config group (’/usr/lib/netdata/conf.d/go.d/apache.conf’): 2 jobs (added: 2, removed: 0)
[ DEBUG ] build[manager] build.go:302 building apache[local] job, config: map[provider:file reader source:/usr/lib/netdata/conf.d/go.d/apache.conf autodetection_retry:0 module:apache name:local priority:70000 update_every:1 url:http://localhost/server-status?auto]
[ DEBUG ] apache[local] apache.go:79 using URL http://localhost/server-status?auto
[ DEBUG ] apache[local] apache.go:80 using timeout: 2s
[ INFO ] apache[local] job.go:171 check success
[ INFO ] build[manager] build.go:208 apache[local] job is being served by another job, skipping it
[ INFO ] apache[local] job.go:191 started, data collection interval 1s
[ DEBUG ] run[manager] run.go:41 tick 0
CHART ‘netdata.execution_time_of_apache_local’ ‘’ ‘Execution time’ ‘ms’ ‘go.d’ ‘netdata.go_plugin_execution_time’ ‘’ ‘145000’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘time’ ‘’ ‘’ ‘1’ ‘1’ ‘’

CHART ‘apache_local.requests’ ‘’ ‘Requests’ ‘requests/s’ ‘requests’ ‘apache.requests’ ‘’ ‘70000’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘total_accesses’ ‘requests’ ‘incremental’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.requests’
SET ‘total_accesses’ = 35520740
END

CHART ‘apache_local.connections’ ‘’ ‘Connections’ ‘connections’ ‘connections’ ‘apache.connections’ ‘’ ‘70001’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘conns_total’ ‘connections’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.connections’
SET ‘conns_total’ =
END

CHART ‘apache_local.conns_async’ ‘’ ‘Async Connections’ ‘connections’ ‘connections’ ‘apache.conns_async’ ‘stacked’ ‘70002’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘conns_async_keep_alive’ ‘keepalive’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘conns_async_closing’ ‘closing’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘conns_async_writing’ ‘writing’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.conns_async’
SET ‘conns_async_keep_alive’ =
SET ‘conns_async_closing’ =
SET ‘conns_async_writing’ =
END

CHART ‘apache_local.scoreboard’ ‘’ ‘Scoreboard’ ‘connections’ ‘connections’ ‘apache.scoreboard’ ‘’ ‘70003’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘scoreboard_waiting’ ‘waiting’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_starting’ ‘starting’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_reading’ ‘reading’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_sending’ ‘sending’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_keepalive’ ‘keepalive’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_dns_lookup’ ‘dns lookup’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_closing’ ‘closing’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_logging’ ‘logging’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_finishing’ ‘finishing’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_idle_cleanup’ ‘idle cleanup’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘scoreboard_open’ ‘open’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.scoreboard’
SET ‘scoreboard_waiting’ = 6
SET ‘scoreboard_starting’ = 0
SET ‘scoreboard_reading’ = 3
SET ‘scoreboard_sending’ = 2
SET ‘scoreboard_keepalive’ = 56
SET ‘scoreboard_dns_lookup’ = 0
SET ‘scoreboard_closing’ = 6
SET ‘scoreboard_logging’ = 0
SET ‘scoreboard_finishing’ = 0
SET ‘scoreboard_idle_cleanup’ = 0
SET ‘scoreboard_open’ = 77
END

CHART ‘apache_local.net’ ‘’ ‘Bandwidth’ ‘kilobits/s’ ‘bandwidth’ ‘apache.net’ ‘area’ ‘70004’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘total_kBytes’ ‘sent’ ‘incremental’ ‘8’ ‘1’ ‘’

BEGIN ‘apache_local.net’
SET ‘total_kBytes’ = 183949352
END

CHART ‘apache_local.workers’ ‘’ ‘Workers Threads’ ‘workers’ ‘workers’ ‘apache.workers’ ‘stacked’ ‘70005’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘idle_workers’ ‘idle’ ‘’ ‘1’ ‘1’ ‘’
DIMENSION ‘busy_workers’ ‘busy’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.workers’
SET ‘idle_workers’ = 6
SET ‘busy_workers’ = 67
END

CHART ‘apache_local.reqpersec’ ‘’ ‘Lifetime Average Number Of Requests Per Second’ ‘requests/s’ ‘statistics’ ‘apache.reqpersec’ ‘area’ ‘70006’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘req_per_sec’ ‘requests’ ‘’ ‘1’ ‘100000’ ‘’

BEGIN ‘apache_local.reqpersec’
SET ‘req_per_sec’ = 2301890
END

CHART ‘apache_local.bytespersec’ ‘’ ‘Lifetime Average Number Of Bytes Served Per Second’ ‘KiB/s’ ‘statistics’ ‘apache.bytespersec’ ‘area’ ‘70007’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘bytes_per_sec’ ‘served’ ‘’ ‘8’ ‘102400000’ ‘’

BEGIN ‘apache_local.bytespersec’
SET ‘bytes_per_sec’ = 12206700000
END

CHART ‘apache_local.bytesperreq’ ‘’ ‘Lifetime Average Response Size’ ‘KiB’ ‘statistics’ ‘apache.bytesperreq’ ‘area’ ‘70008’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘bytes_per_req’ ‘size’ ‘’ ‘1’ ‘102400000’ ‘’

BEGIN ‘apache_local.bytesperreq’
SET ‘bytes_per_req’ = 530293000
END

CHART ‘apache_local.uptime’ ‘’ ‘Uptime’ ‘seconds’ ‘availability’ ‘apache.uptime’ ‘’ ‘70009’ ‘1’ ‘’ ‘go.d’ ‘apache’
DIMENSION ‘uptime’ ‘’ ‘’ ‘1’ ‘1’ ‘’

BEGIN ‘apache_local.uptime’
SET ‘uptime’ = 1543115
END

BEGIN ‘netdata.execution_time_of_apache_local’
SET ‘time’ = 0
END

[ DEBUG ] run[manager] run.go:41 tick 1
BEGIN ‘apache_local.requests’ 1000855
SET ‘total_accesses’ = 35520786
END

BEGIN ‘apache_local.connections’
SET ‘conns_total’ =
END

BEGIN ‘apache_local.conns_async’
SET ‘conns_async_keep_alive’ =
SET ‘conns_async_closing’ =
SET ‘conns_async_writing’ =
END

BEGIN ‘apache_local.scoreboard’ 1000855
SET ‘scoreboard_waiting’ = 7
SET ‘scoreboard_starting’ = 0
SET ‘scoreboard_reading’ = 1
SET ‘scoreboard_sending’ = 1
SET ‘scoreboard_keepalive’ = 53
SET ‘scoreboard_dns_lookup’ = 0
SET ‘scoreboard_closing’ = 11
SET ‘scoreboard_logging’ = 0
SET ‘scoreboard_finishing’ = 0
SET ‘scoreboard_idle_cleanup’ = 0
SET ‘scoreboard_open’ = 77
END

BEGIN ‘apache_local.net’ 1000855
SET ‘total_kBytes’ = 183949489
END

BEGIN ‘apache_local.workers’ 1000855
SET ‘idle_workers’ = 7
SET ‘busy_workers’ = 66
END

BEGIN ‘apache_local.reqpersec’ 1000855
SET ‘req_per_sec’ = 2301890
END

BEGIN ‘apache_local.bytespersec’ 1000855
SET ‘bytes_per_sec’ = 12206700000
END

BEGIN ‘apache_local.bytesperreq’ 1000855
SET ‘bytes_per_req’ = 530293000
END

BEGIN ‘apache_local.uptime’ 1000855
SET ‘uptime’ = 1543116
END

BEGIN ‘netdata.execution_time_of_apache_local’ 1000855
SET ‘time’ = 1
END

4

I don’t use local pages at all so I can’t tell if the local monitoring page would be showing it.

I am not sure what steps you followed to restrict access to the dashboard, could you verify that we can see the charts in question under <node_ip>:19999/api/v1/charts?

If yes then this is a Netdata cloud issue.

5

I did this: curl localhost:19999/api/v1/charts | grep apache_local.workers on both a working and non-working node and could see similar output from the two, here is the complete entry from the “non-working” node:

            "apache_local.workers":                 {
                    "id": "apache_local.workers",
                    "name": "apache_local.workers",
                    "type": "apache_local",
                    "family": "workers",
                    "context": "apache.workers",
                    "title": "Workers Threads (apache_local.workers)",
                    "priority": 70005,
                    "plugin": "go.d",
                    "module": "apache",
                    "units": "workers",
                    "data_url": "/api/v1/data?chart=apache_local.workers",
                    "chart_type": "stacked",
                    "duration": 1545094,
                    "first_entry": 1651764164,
                    "last_entry": 1653309257,
                    "update_every": 1,
                    "dimensions": {
                            "idle_workers": { "name": "idle" },
                            "busy_workers": { "name": "busy" }
                    },
                    "chart_variables": {
                    },
                    "green": null,
                    "red": null,
                    "alarms": {

                    },
                    "chart_labels": {

                    }

            },
6

Opened an upstream issue [Bug]: Missing charts on Netdata Cloud · Issue #437 · netdata/netdata-cloud · GitHub you can follow the progress there.

7

While investigating I noticed there is a similar issue with apache_local.scoreboard, 3 nodes are showing it out of 5, but it’s not the same nodes!
workers are shown on nodes 1, 2 and 5 and it’s listed in all servers v1/charts URL.
scoreboard is shown on nodes 1, 4 and 5 and it’s listed in all servers but 2! ?

Really odd.

PS: Thanks for opening the issue.

1 Like