Any idea how to avoid this issue?
We’re running multiple machines with apache, but this one that has shared hosting on it, is constantly throwing log errors which we don’t understand.
Welcome back friend,
So, this alarm says that it has been <alarm_value> (370569) seconds since the last time which it managed to collect data from the data source, in this case the apache server.
@joel has recently released a video about creating and configuring the alarms on the Netdata
Agent, so it might prove useful:
Moreover, we also recently published a blog post about the structure of our alarms and what all these fields mean:
https://www.netdata.cloud/blog/product/actionable-intelligent-alerts/
If you have any further questions I will be happy to assist. Alarms is an area which we know we can improve, so you can be certain that we are working on improving the experience!
But why can’t it collect data for Apache on this specific machine?
The other logs like access log and other vhosts log are working fine.
It seems the Apache local monitoring stopped at 5 of february for some reason?
Which log file is it looking at in this case? How can i see which files the monitor checks?
So, let’s try debug this. It doesn’t sound right.
For starters, we need the error.log file that exists in /var/log/netdata/error.log. Can you grep apache that file and copy-paste output?
There’s basically no errors in that log file, only this:
2021-02-11 12:36:31: tc-qos-helper.sh: WARNING: FireQoS is not installed on this system. Use FireQoS to apply traffic QoS and expose the class names to netdata. Check https://github.com/$
2021-02-11 12:36:31: tc-qos-helper.sh: WARNING: Cannot find file '/usr/lib/netdata/conf.d/tc-qos-helper.conf'.
2021-02-11 12:36:31: tc-qos-helper.sh: WARNING: Cannot find file '/etc/netdata/tc-qos-helper.conf'.
I did find this in an older log though (02-10):
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests' on host 'server' is 5.53519 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.excluded_requests' on host 'server' is 5.53545 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_vhost' on host 'server' is 5.53546 secs in the past (counter #961882, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_port' on host 'server' is 5.53551 secs in the past (counter #961883, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_ip_proto' on host 'server' is 5.53552 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.current_poll_uniq_clients' on host 'server' is 5.53552 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_http_method' on host 'server' is 5.53554 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_http_version' on host 'server' is 5.53553 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_type' on host 'server' is 5.53553 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.responses_by_status_code_class' on host 'server' is 5.53552 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_2xx_responses' on host 'server' is 5.53552 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_3xx_responses' on host 'server' is 5.53552 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_4xx_responses' on host 'server' is 5.53551 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_5xx_responses' on host 'server' is 5.53549 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.bandwidth' on host 'server' is 5.53545 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:56: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'netdata.execution_time_of_web_log_apache_vhosts' on host 'server' is 5.53544 secs in the past (counter #961880, update #961853). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests' on host 'server' is 3.46383 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.excluded_requests' on host 'server' is 3.46400 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_vhost' on host 'server' is 3.46401 secs in the future (counter #961893, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_port' on host 'server' is 3.46342 secs in the future (counter #961894, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_ip_proto' on host 'server' is 3.46253 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.current_poll_uniq_clients' on host 'server' is 3.46251 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_http_method' on host 'server' is 3.46253 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_http_version' on host 'server' is 3.46253 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.requests_by_type' on host 'server' is 3.46253 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.responses_by_status_code_class' on host 'server' is 3.46248 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_2xx_responses' on host 'server' is 3.45087 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_3xx_responses' on host 'server' is 3.45077 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_4xx_responses' on host 'server' is 3.45074 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.status_code_class_5xx_responses' on host 'server' is 3.45074 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'web_log_apache_vhosts.bandwidth' on host 'server' is 3.45073 secs in the future (counter #961891, update #961855). Adjusting it to current time.
2021-02-10 16:37:58: netdata INFO : PLUGINSD[go.d] : RRD database for chart 'netdata.execution_time_of_web_log_apache_vhosts' on host 'server' is 3.45073 secs in the future (counter #961891, update #961855). Adjusting it to current time.
I checked the current machine time and it’s correct to the second.
1 Like
Have you changed the memory mode for your netdata installations or are you using the default values?
Pinging @ilyam8 who is our integrations chief.
Hi, guys.
apache_last_collected_secs alarm checks apache collector is up and running (apache.requests chart is being updated).
Those charts are from web_log collector.
By default apache collects metrics from http://127.0.0.1/server-status?auto. It seems it was, but no longer able to do it for some reason.
@tmikaeld do you see Apache section on the dashboard?
Apache mod_status ( server-status) was leaking request IPs and URIs publicly so it has been disabled, I really wouldn’t rely on that by default as it is even deemed as a vulnerability by some security researchers.
From that link:
Summary:
Requesting the URI /server-status provides information on the server activity and performance.
It provides, indeed. That info allows a server administrator to find out how well their server is performing.
I suggest to use Require directive to restrict access to the server-status.
Example:
<Location "/server-status">
SetHandler server-status
Require ip 127.0.0.1
</Location>
1 Like
Back to the OP
Apache_last_collected_secs errors constantlycollected-secs-errors-constantly
Netdata Agent restart resolves the issue.
And I hope Netdata doesn’t expose sensitive information like IPs and URIs?
Netdata has web API, you can query all the charts you see on the dashboard. Ensure it is not public available and you are good.
By default netdata installs on port 19999 publicly, you mean there’s sensitive data on that dashboard?
Check the API, for instance /api/v1/info endpoint returns a lot of info about the system. Perhaps it is considered as a sensitive info in your organization.
19999 publicly
Yeah, i believe it is 0.0.0.0:19999. You can restrict access by using Access Lists feature. Another options are: using firewall, bind netdata to 127.0.0.1 and use NGINX as reverse proxy (or Apache if you are using it already).
I’m using the Netdata cloud to monitor multiple hosts at the same time, if I block the public port then Netdata cloud stops working (naturally). How do you suggest I make them work while still not being publicly available?
Netdata Cloud doesn’t require any incoming/listening open port. ACLK/Netdata Cloud only makes an outbound connection to https://app.netdata.cloud which is then upgraded to WebSockets.
Maybe it is terminology problem here?
Netdata Cloud is what you access trough https://app.netdata.cloud to have central monitoring of multiple nodes.
When you are talking about port 19999 which is local dashboard of that node (and potentially nodes streaming to it) yes if you block that you can’t access it. It has nothing to do with Netdata Cloud though. You should be able to block local dashboard completely and use only Netdata Cloud to access charts. As said previously local agent → netdata cloud is outbound connection though (port 443 app.netdata.cloud). You can configure it to also use SOCKS5 or HTTP proxy if you need to do so.
I tried blocking 19999 and it still works on the netdata cloud, was this always the case? Or a recent change?
1 Like
@joel to the rescue, with a blog post he wrote a few months back:
In general, when having a question, search our blog first (Blog | Netdata - Netdata) and then our documentation portal (https://learn.netdata.cloud). The final destination should be our community(https://community.netdata.cloud).