In the Logs tab I see a lot of stuff coming from the syslog (I think?) but not the web log from Nginx. This makes it hard to solve things like the web_log_1m_redirects alert, without ssh’ing into my server and digging through the logs myself. If the logs were visible in Netdata and I could filter them on time and perhaps status code that would make my life a lot easier.
Is this possible?
Hi @Kevin_Renskers ,
We actually covered that question in our last Office Hours episode, if you want to the check the slide summarizing that or the actual documentation on Learn.
Cheers,
I followed all the steps from log2journal | Learn Netdata, and this is what I am now seeing in the Logs tab:
The screenshot from that docs article looks a lot more useful:
How do I get the data to show up like that? Just following the steps definitely doesn’t result in this.
I also have to say that this seems really rather complex just to get web logs to show up in this tab. Can’t the Netdata software on my server automate this? It already knows Nginx is running, all the Nginx metrics show up and I didn’t have to do anything. Why is getting access to the logs not automatically done as well?
How do I get the data to show up like that? Just following the steps definitely doesn’t result in this.
Are you referring to the displayed column, the missing chart?
If if it is the displayed columns this can be achieved by going into the cogwheel above the table on the top right corner and select the columns to display
Note: Atm it isn’t yet possible to save these column selection as a saved view but it is something we plan to bring as customization settings
If it is the missing chart I would suggest for you to fill in a bug report.
I also have to say that this seems really rather complex just to get web logs to show up in this tab. Can’t the Netdata software on my server automate this? It already knows Nginx is running, all the Nginx metrics show up and I didn’t have to do anything. Why is getting access to the logs not automatically done as well?
Netdata’s strongest point is on the metrics collection and out-of-the-box dashboards and alerts. We have recently started moving towards logs and we are building integrations with systems that allow us to surfaces logs, first is was systemd journal but we also want to integrate with Loki and Elasticsearch.
Having said this, Netdata doesn’t collect and store your logs and we don’t know to which logging system your pushing your logs to. We will of course keep an eye on such feedback to try to understand where we can improve the user experience and bring the best out-of-the-box experience, like in metrics.
I meant the columns, not the chart (I just didn’t include that in my screenshot). When I press the cogwheel I see nothing related to nginx that I can select. NGINX_METHOD and NGINX_STATUS that are shown in the docs’ screenshot, that’s not a column I can choose.
If the fields are passed on to the journal like this example log2journal | Learn Netdata
NGINX_BODY_BYTES_SENT=4172
NGINX_HTTP_REFERER=-
NGINX_HTTP_USER_AGENT=Go-http-client/1.1
NGINX_REMOTE_ADDR=1.2.3.4
NGINX_REMOTE_USER=-
NGINX_REQUEST_METHOD=GET
NGINX_REQUEST_URI=/index.html
NGINX_SERVER_PROTOCOL=HTTP/1.1
NGINX_STATUS=200
they should be available as columns and for to see see them as selectors on the filters.
Will try to share this request internally and see if someone from the team can help troubleshoot this.
Could you share your log2journal configurations and few log lines as examples?
I’m using literally the same file as that learn article:
pattern: |
(?x) # Enable PCRE2 extended mode
^
(?<remote_addr>[^ ]+) \s - \s
(?<remote_user>[^ ]+) \s
\[
(?<time_local>[^\]]+)
\]
\s+ "
(?<request>
(?<request_method>[A-Z]+) \s+
(?<request_uri>[^ ]+) \s+
(?<server_protocol>[^"]+)
)
" \s+
(?<status>\d+) \s+
(?<body_bytes_sent>\d+) \s+
"(?<http_referer>[^"]*)" \s+
"(?<http_user_agent>[^"]*)"
prefix: 'NGINX_'
rename:
- new_key: MESSAGE
old_key: NGINX_REQUEST
inject:
- key: PRIORITY
value: '${NGINX_STATUS}'
- key: SYSLOG_IDENTIFIER
value: 'nginx-log'
rewrite:
- key: PRIORITY
match: '^[123]'
value: 6
- key: PRIORITY
match: '^4'
value: 5
- key: PRIORITY
match: '^5'
value: 3
- key: PRIORITY
match: '.*'
value: 4
Some lines from Nginx’s access log:
104.28.105.10 - - [15/Apr/2024:17:12:32 +0000] "GET /shop/basket/ HTTP/2.0" 200 54 "https://www.soundradix.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15"
104.28.105.10 - - [15/Apr/2024:17:12:32 +0000] "GET /shop/owned/products/ HTTP/2.0" 200 2 "https://www.soundradix.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15"
104.28.105.10 - - [15/Apr/2024:17:12:32 +0000] "GET /shop/owned/redeemcodes/ HTTP/2.0" 200 2 "https://www.soundradix.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15"
104.28.105.10 - - [15/Apr/2024:17:12:32 +0000] "GET /shop/products/releases/surfereq/ HTTP/2.0" 200 4734 "https://www.soundradix.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15"
104.28.105.10 - - [15/Apr/2024:17:12:32 +0000] "GET /shop/products/downloads/ HTTP/2.0" 200 1445 "https://www.soundradix.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15"
104.28.105.10 - - [15/Apr/2024:17:12:32 +0000] "GET /shop/offers/ HTTP/2.0" 200 627 "https://www.soundradix.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15"
5.161.49.218 - - [15/Apr/2024:17:12:38 +0000] "GET /flags/ HTTP/1.1" 200 145 "-" "node"
5.161.49.218 - - [15/Apr/2024:17:12:38 +0000] "GET /shop/products/ HTTP/1.1" 200 7508 "-" "node"
Was someone from the team able to troubleshoot this?