Well, the config looks valid 
Is it only 3 lines in there? Can you just cat and show the output?
btw how did you install Netdata? `
The following command output would be helpful too
ps faxu | grep "[n]etdata"
The following is the whole config but i sent you only the uncomment.
Is there other module or config which is compatible with nginx plus?
# netdata go.d.plugin configuration for web_log
#
# This file is in YAML format. Generally the format is:
#
# name: value
#
# There are 2 sections:
# - GLOBAL
# - JOBS
#
#
# [ GLOBAL ]
# These variables set the defaults for all JOBs, however each JOB may define its own, overriding the defaults.
#
# The GLOBAL section format:
# param1: value1
# param2: value2
#
# Currently supported global parameters:
# - update_every
# Data collection frequency in seconds. Default: 1.
#
# - autodetection_retry
# Re-check interval in seconds. Attempts to start the job are made once every interval.
# Zero means not to schedule re-check. Default: 0.
#
# - priority
# Priority is the relative priority of the charts as rendered on the web page,
# lower numbers make the charts appear before the ones with higher numbers. Default: 70000.
#
#
# [ JOBS ]
# JOBS allow you to collect values from multiple sources.
# Each source will have its own set of charts.
#
# IMPORTANT:
# - Parameter 'name' is mandatory.
# - Jobs with the same name are mutually exclusive. Only one of them will be allowed running at any time.
#
# This allows auto detection to try several alternatives and pick the one that works.
# Any number of jobs is supported.
#
# The JOBS section format:
#
# jobs:
# - name: job1
# param1: value1
# param2: value2
#
# - name: job2
# param1: value1
# param2: value2
#
# - name: job2
# param1: value1
#
#
# [ List of JOB specific parameters ]:
# - path
# The path to web server log file, can use wildcard.
# Syntax:
# path: /path/to/log/file
# path: /path/to/log/*.log
#
# - exclude_path
# The path to be excluded, can use wildcard.
# Syntax:
# exclude_path: *.tar.gz
#
# - url_patterns
# Requests per URL pattern chart. Matches against URL field.
# Matcher pattern syntax: https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format
# Syntax:
# url_patterns:
# - name: name1 # Name will be used as dimension.
# match: pattern # Pattern syntax: matcher.
# - name: name2
# match: pattern
#
# - custom_fields
# Requests per pattern charts. Matches against <custom> fields. Used in custom log format.
# Matcher pattern syntax: https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format
# Syntax:
# custom_fields:
# - name: field_name1 # Field name. Should match field name from log format.
# patterns:
# - name: name1 # Name will be used as dimension.
# match: pattern # Pattern syntax: matcher.
# - name: name2
# match: pattern
# - name: field_name2
# patterns:
# - name: name1
# match: pattern
# - name: name2
# match: pattern
#
# - custom_time_fields
# Count min/avg/max and also cumulative histogram for defined custom time fields like apache LogIOTrackTTFB. Used in custom log format.
# Syntax:
# custom_time_fields:
# - name: field_name1 # Field name. Should match field name from log format.
# histogram: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10] # optional field
# - name: field_name2
# histogram: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
#
# - histogram
# Cumulative histogram of response time in seconds.
# Syntax:
# histogram: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
#
# - group_response_codes
# Group response codes by code class (informational, successful, redirects, client and server errors).
# Syntax:
# group_response_codes: yes/no
#
# - log_type
# One of supported log types: csv, ltsv, regexp, auto.
# If set to auto module will try to auto-detect log type and format.
# Auto-detection order: ltsv, csv.
# Syntax:
# log_type: auto/csv/ltsv/regexp
#
# - csv_config
# CSV log type specific parameters.
# Syntax:
# csv_config:
# format: 'format' # Defines log format. You need define nginx or apache log format.
# Nginx: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
# Apache: http://httpd.apache.org/docs/current/mod/mod_log_config.html
# fields_per_record: -1 # The number of expected fields per record. If set to positive, csv parser requires
# each record to have the given number of fields. If set to 0, csv parser sets it to
# the number of fields in the first record, so that future records must have the same
# field count. If fields_per_record is negative, no check is made and records may have
# a variable number of fields.
# delimiter: ' ' # Field delimiter.
# trim_leading_space: yes/no # If set to true, leading white space in a field is ignored.
#
# - ltsv_config
# LTSV log type specific parameters.
# Syntax:
# ltsv_config:
# field_delimiter: ' ' # Fields delimiter.
# value_delimiter: ':' # Label-value pairs delimiter.
# mapping: # Label field mapping.
# label1: field1
# label2: field2
#
# - json_config
# JSON log type specific parameters.
# Syntax:
# json_config:
# mapping: # Label field mapping, json-log-label: weblog-label
# label1: field1
# label2: field2
#
# - regexp_config
# RegExp log type specific parameters.
# Pattern syntax: https://golang.org/pkg/regexp/syntax/.
# Syntax:
# regexp_config:
# pattern: pattern # Pattern. It is mandatory to have subexpressions names in the pattern.
#
#
# [ JOB defaults ]:
# exclude_path: *.gz
# group_response_codes: yes
# log_type: auto
# csv_config:
# fields_per_record: -1
# delimiter: ' '
# trim_leading_space: no
# ltsv_config:
# field_delimiter: ' '
# value_delimiter: ':'
#
#
# [ JOB mandatory parameters ]:
# - name
# - path
#
# ------------------------------------------------MODULE-CONFIGURATION--------------------------------------------------
# [ GLOBAL ]
# update_every: 10
autodetection_retry: 30
# priority: 70000
#
#
# [ JOBS ]
jobs:
# NGINX
# debian, arch
- name: nginx_master
path: /var/log/nginx/master.access.log
# log_type: auto
# # gentoo
# - name: nginx
# path: /var/log/nginx/localhost.access_log
# # APACHE
# # debian
# - name: apache
# path: /var/log/apache2/access.log
# # gentoo
# - name: apache
# path: /var/log/apache2/access_log
# # arch
# - name: apache
# path: /var/log/httpd/access_log
# # debian
# - name: apache_vhosts
# path: /var/log/apache2/other_vhosts_access.log
# # GUNICORN
# - name: gunicorn
# path: /var/log/gunicorn/access.log
# - name: gunicorn
# path: /var/log/gunicorn/gunicorn-access.log
Netdata installation was done with the following packages and signing key
https://packagecloud.io/netdata/netdata/gpgkey
deb https://packagecloud.io/netdata/netdata/ubuntu/ focal main
netdata 1621182 2.4 0.5 362588 83172 ? Ssl 13:11 0:04 /usr/sbin/netdata -D
netdata 1621186 0.0 0.0 32064 6540 ? Sl 13:11 0:00 \_ /usr/sbin/netdata --special-spawn-server
netdata 1621347 0.1 0.0 4248 3324 ? S 13:11 0:00 \_ bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
netdata 1621353 1.5 0.0 54204 14344 ? S 13:11 0:03 \_ /usr/libexec/netdata/plugins.d/apps.plugin 1
netdata 1621363 0.3 0.1 723664 20636 ? Sl 13:11 0:00 \_ /usr/libexec/netdata/plugins.d/go.d.plugin 1
root 1621364 0.9 0.0 453876 3524 ? Sl 13:11 0:01 \_ /usr/libexec/netdata/plugins.d/ebpf.plugin 1
root 1621365 0.0 0.0 5128 912 ? S 13:11 0:00 \_ /usr/libexec/netdata/plugins.d/nfacct.plugin 1
After investigation, i tried to add job in web_log.conf but from debug mode i see errors to parse csv.
[ DEBUG ] main[main] main.go:111 plugin: name=go.d, version=v0.28.2
[ DEBUG ] main[main] main.go:113 current user: name=netdata, uid=116
[ INFO ] main[main] agent.go:106 instance is started
[ INFO ] main[main] setup.go:39 loading config file
[ INFO ] main[main] setup.go:47 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d]
[ INFO ] main[main] setup.go:54 found '/etc/netdata/go.d.conf
[ INFO ] main[main] setup.go:61 config successfully loaded
[ INFO ] main[main] agent.go:110 using config: enabled 'true', default_run 'true', max_procs '0'
[ INFO ] main[main] setup.go:66 loading modules
[ INFO ] main[main] setup.go:85 enabled/registered modules: 1/59
[ INFO ] main[main] setup.go:90 building discovery config
[ INFO ] main[main] setup.go:116 looking for 'web_log.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[ INFO ] main[main] setup.go:123 found '/etc/netdata/go.d/web_log.conf
[ INFO ] main[main] setup.go:128 dummy/read/watch paths: 0/1/0
[ INFO ] discovery[manager] manager.go:90 registered discoverers: [file discovery: [file reader]]
[ INFO ] discovery[manager] manager.go:95 instance is started
[ INFO ] discovery[file manager] discovery.go:71 instance is started
[ INFO ] discovery[file reader] read.go:39 instance is started
[ INFO ] build[manager] build.go:106 instance is started
[ INFO ] run[manager] run.go:30 instance is started
[ INFO ] discovery[file reader] read.go:40 instance is stopped
[ DEBUG ] build[manager] build.go:153 received config group ('/etc/netdata/go.d/web_log.conf'): 1 jobs (added: 1, removed: 0)
[ DEBUG ] build[manager] build.go:302 building web_log[nginx_master] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/web_log.conf autodetection_retry:30 log_type:auto module:web_log name:nginx_master path:/var/log/nginx/master.access.log priority:70000 update_every:1]
[ DEBUG ] web_log[nginx_master] init.go:31 skipping URL patterns creating, no patterns provided
[ DEBUG ] web_log[nginx_master] init.go:49 skipping custom fields creating, no custom fields provided
[ DEBUG ] web_log[nginx_master] init.go:74 skipping custom time fields creating, no custom time fields provided
[ DEBUG ] web_log[nginx_master] init.go:103 starting log reader creating
[ DEBUG ] web_log[nginx_master] reader.go:71 open log file: /var/log/nginx/master.access.log
[ DEBUG ] web_log[nginx_master] init.go:108 created log reader, current file '/var/log/nginx/master.access.log'
[ DEBUG ] web_log[nginx_master] init.go:114 starting parser creating
[ DEBUG ] web_log[nginx_master] init.go:120 last line: '- - - [29/Oct/2021:16:27:43 +0300] TLSv1.3/TLS_AES_256_GCM_SHA384 "GET /path/to/something HTTP/1.1" 200 1456 "HOST" "https://host_referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36" "-" "-" "W3" rt=0.019 uct="0.012" urt="0.020"'
[ DEBUG ] web_log[nginx_master] parser.go:75 log_type is auto, will try format auto-detection
[ DEBUG ] web_log[nginx_master] parser.go:98 starting log type auto-detection
[ DEBUG ] web_log[nginx_master] parser.go:107 log type is CSV
[ DEBUG ] web_log[nginx_master] parser.go:112 starting csv log format auto-detection
[ DEBUG ] web_log[nginx_master] parser.go:113 config: {FieldsPerRecord:-1 Delimiter: TrimLeadingSpace:false Format: CheckField:0xbca280}
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ DEBUG ] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent'
[ DEBUG ] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[ WARN ] web_log[nginx_master] weblog.go:112 check failed: create parser: cannot auto-detect log format, use custom log format
[ ERROR ] web_log[nginx_master] job.go:157 check failed
[ DEBUG ] web_log[nginx_master] reader.go:127 close log file: /var/log/nginx/master.access.log
[ INFO ] build[manager] build.go:263 web_log[nginx_master] job detection failed, will retry in 30 seconds
Hmm, I was sure you had had it in there
Yes, the plugin couldn’t detect the log format (tried predefined patterns and failed). Could you show a logline and log_format from your nginx.conf?
Oh, I see
- - - [29/Oct/2021:16:27:43 +0300] TLSv1.3/TLS_AES_256_GCM_SHA384 "GET /path/to/something HTTP/1.1" 200 1456 "HOST" "https://host_referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36" "-" "-" "W3" rt=0.019 uct="0.012" urt="0.020"
Yes this is a logline but i change only the http_referrer and the host for security reasons.
The following is one of nginx log_format
log_format '$http_true_client_ip - $remote_user [$time_local] '
'$ssl_protocol/$ssl_cipher '
'"$request" $status $body_bytes_sent '
'"$host" "$http_referer" "$http_user_agent" "$http_x_request_id" "$correlationId" "$sent_http_prx" '
'rt=$request_time uct="$upstream_connect_time" urt="$upstream_response_time"';
I was wondering why in a server with nginx opensource and netdata is working properly.
To be more specific, in the other server with opensource nginx it works with python plugin web log , but now i tried with both plugins and i see the same result.
Is there any issue with web_log config in plugins and nginx plus?
Is there any idea or other configuration (custom format for the specific?) for this setup?
@nmarantos There is a list of predefined patterns (works for nginx/apache), here it is (go version, but the same logic applies for py): https://github.com/netdata/go.d.plugin/blob/83f00839d2675d308dddc0a5f2cf02e999cbe06f/modules/weblog/parser.go#L36-L45
The plugin tries all of them and if none works you get
[ WARN ] web_log[nginx_master] weblog.go:112 check failed: create parser: cannot auto-detect log format, use custom log format
Is the log_format you shared default? Or did you change it?
Ηι @ilyam8 ,
$http_true_client_ip $ssl_protocol/$ssl_cipher "$http_referer" "$http_user_agent" "$http_x_request_id" "$upstreamCorrelationId" "$sent_http_prx"
These were added in the log format which i use, but i had not any issue in previous server with nginx opensource and python.d plugin.
UPDATE
I changed log format like the last predefined and it works.
How can i add the above fields in the log format i want?
The log format with which it played and i want fields to be added to it.
$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time
There is a list of known fields. All of them can be used when creating a custom pattern.
A note: a space is a default separator when using CSV format. So
should be $ssl_protocol $ssl_cipher (if we are interested in SSL metrics/charts).
Python version uses regular expressions (see the list of predefined regular expressions).
Go version supports the following parsers:
- CSV (default)
- JSON
- RegExp (python uses only this)
- LTSV
So you are going to use this format, correct?
Do you remember what was the default Nginx plus format?
Hi @ilyam8,
From nginx.conf i changed my log format to one line and in web_log.conf in custom csv_config i configured it with the same raw and the same type. Maybe before some variable did not recognize it or was not in the same order.
Now with this way, it is working.