# Web\_log.conf for nginx

**URL:** <https://community.netdata.cloud/t/web-log-conf-for-nginx/39>\
**Category:** Help\
**Tags:** agent, agent-collectors\
**Created:** [November 2, 2020, 1:58pm UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39 "2020-11-02T13:58:55Z")\
**Posts on this page:** 9\
**Page:** 2

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [October 29, 2021, 9:56am UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/21 "2021-10-29T09:56:12Z")

</div>

Well, the config looks valid 🤔 Is it only 3 lines in there? Can you just `cat` and show the output?

btw how did you install Netdata? `

The following command output would be helpful too

```nohighlight
ps faxu | grep "[n]etdata"

```

---

<div class="post-metadata">

**Author:** ![nmarantos](https://avatars.discourse-cdn.com/v4/letter/n/898d66/32.png) [@nmarantos](https://community.netdata.cloud/u/nmarantos)\
**Post date:** [October 29, 2021, 10:21am UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/22 "2021-10-29T10:21:20Z")

</div>

The following is the whole config but i sent you only the uncomment.  
Is there other module or config which is compatible with nginx plus?

```auto
# netdata go.d.plugin configuration for web_log
#
# This file is in YAML format. Generally the format is:
#
# name: value
#
# There are 2 sections:
# - GLOBAL
# - JOBS
#
#
# [GLOBAL]
# These variables set the defaults for all JOBs, however each JOB may define its own, overriding the defaults.
#
# The GLOBAL section format:
# param1: value1
# param2: value2
#
# Currently supported global parameters:
# - update_every
# Data collection frequency in seconds. Default: 1.
#
# - autodetection_retry
# Re-check interval in seconds. Attempts to start the job are made once every interval.
# Zero means not to schedule re-check. Default: 0.
#
# - priority
# Priority is the relative priority of the charts as rendered on the web page,
# lower numbers make the charts appear before the ones with higher numbers. Default: 70000.
#
#
# [JOBS]
# JOBS allow you to collect values from multiple sources.
# Each source will have its own set of charts.
#
# IMPORTANT:
# - Parameter 'name' is mandatory.
# - Jobs with the same name are mutually exclusive. Only one of them will be allowed running at any time.
#
# This allows auto detection to try several alternatives and pick the one that works.
# Any number of jobs is supported.
#
# The JOBS section format:
#
# jobs:
# - name: job1
# param1: value1
# param2: value2
#
# - name: job2
# param1: value1
# param2: value2
#
# - name: job2
# param1: value1
#
#
# [List of JOB specific parameters]:
# - path
# The path to web server log file, can use wildcard.
# Syntax:
# path: /path/to/log/file
# path: /path/to/log/*.log
#
# - exclude_path
# The path to be excluded, can use wildcard.
# Syntax:
# exclude_path: *.tar.gz
#
# - url_patterns
# Requests per URL pattern chart. Matches against URL field.
# Matcher pattern syntax: https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format
# Syntax:
# url_patterns:
# - name: name1 # Name will be used as dimension.
# match: pattern # Pattern syntax: matcher.
# - name: name2
# match: pattern
#
# - custom_fields
# Requests per pattern charts. Matches against <custom> fields. Used in custom log format.
# Matcher pattern syntax: https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format
# Syntax:
# custom_fields:
# - name: field_name1 # Field name. Should match field name from log format.
# patterns:
# - name: name1 # Name will be used as dimension.
# match: pattern # Pattern syntax: matcher.
# - name: name2
# match: pattern
# - name: field_name2
# patterns:
# - name: name1
# match: pattern
# - name: name2
# match: pattern
#
# - custom_time_fields
# Count min/avg/max and also cumulative histogram for defined custom time fields like apache LogIOTrackTTFB. Used in custom log format.
# Syntax:
# custom_time_fields:
# - name: field_name1 # Field name. Should match field name from log format.
# histogram: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10] # optional field
# - name: field_name2
# histogram: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
#
# - histogram
# Cumulative histogram of response time in seconds.
# Syntax:
# histogram: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
#
# - group_response_codes
# Group response codes by code class (informational, successful, redirects, client and server errors).
# Syntax:
# group_response_codes: yes/no
#
# - log_type
# One of supported log types: csv, ltsv, regexp, auto.
# If set to auto module will try to auto-detect log type and format.
# Auto-detection order: ltsv, csv.
# Syntax:
# log_type: auto/csv/ltsv/regexp
#
# - csv_config
# CSV log type specific parameters.
# Syntax:
# csv_config:
# format: 'format' # Defines log format. You need define nginx or apache log format.
# Nginx: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
# Apache: http://httpd.apache.org/docs/current/mod/mod_log_config.html
# fields_per_record: -1 # The number of expected fields per record. If set to positive, csv parser requires
# each record to have the given number of fields. If set to 0, csv parser sets it to
# the number of fields in the first record, so that future records must have the same
# field count. If fields_per_record is negative, no check is made and records may have
# a variable number of fields.
# delimiter: ' ' # Field delimiter.
# trim_leading_space: yes/no # If set to true, leading white space in a field is ignored.
#
# - ltsv_config
# LTSV log type specific parameters.
# Syntax:
# ltsv_config:
# field_delimiter: ' ' # Fields delimiter.
# value_delimiter: ':' # Label-value pairs delimiter.
# mapping: # Label field mapping.
# label1: field1
# label2: field2
#
# - json_config
# JSON log type specific parameters.
# Syntax:
# json_config:
# mapping: # Label field mapping, json-log-label: weblog-label
# label1: field1
# label2: field2
#
# - regexp_config
# RegExp log type specific parameters.
# Pattern syntax: https://golang.org/pkg/regexp/syntax/.
# Syntax:
# regexp_config:
# pattern: pattern # Pattern. It is mandatory to have subexpressions names in the pattern.
#
#
# [JOB defaults]:
# exclude_path: *.gz
# group_response_codes: yes
# log_type: auto
# csv_config:
# fields_per_record: -1
# delimiter: ' '
# trim_leading_space: no
# ltsv_config:
# field_delimiter: ' '
# value_delimiter: ':'
#
#
# [JOB mandatory parameters]:
# - name
# - path
#
# ------------------------------------------------MODULE-CONFIGURATION--------------------------------------------------
# [GLOBAL]
# update_every: 10
autodetection_retry: 30
# priority: 70000
#
#
# [JOBS]
jobs:
  # NGINX
  # debian, arch
  - name: nginx_master
    path: /var/log/nginx/master.access.log
# log_type: auto

# # gentoo
# - name: nginx
# path: /var/log/nginx/localhost.access_log

# # APACHE
# # debian
# - name: apache
# path: /var/log/apache2/access.log

# # gentoo
# - name: apache
# path: /var/log/apache2/access_log

# # arch
# - name: apache
# path: /var/log/httpd/access_log

# # debian
# - name: apache_vhosts
# path: /var/log/apache2/other_vhosts_access.log

# # GUNICORN
# - name: gunicorn
# path: /var/log/gunicorn/access.log

# - name: gunicorn
# path: /var/log/gunicorn/gunicorn-access.log

```

Netdata installation was done with the following packages and signing key

```auto
https://packagecloud.io/netdata/netdata/gpgkey
deb https://packagecloud.io/netdata/netdata/ubuntu/ focal main

```

```auto
netdata 1621182 2.4 0.5 362588 83172 ? Ssl 13:11 0:04 /usr/sbin/netdata -D
netdata 1621186 0.0 0.0 32064 6540 ? Sl 13:11 0:00 \_ /usr/sbin/netdata --special-spawn-server
netdata 1621347 0.1 0.0 4248 3324 ? S 13:11 0:00 \_ bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
netdata 1621353 1.5 0.0 54204 14344 ? S 13:11 0:03 \_ /usr/libexec/netdata/plugins.d/apps.plugin 1
netdata 1621363 0.3 0.1 723664 20636 ? Sl 13:11 0:00 \_ /usr/libexec/netdata/plugins.d/go.d.plugin 1
root 1621364 0.9 0.0 453876 3524 ? Sl 13:11 0:01 \_ /usr/libexec/netdata/plugins.d/ebpf.plugin 1
root 1621365 0.0 0.0 5128 912 ? S 13:11 0:00 \_ /usr/libexec/netdata/plugins.d/nfacct.plugin 1

```

---

<div class="post-metadata">

**Author:** ![nmarantos](https://avatars.discourse-cdn.com/v4/letter/n/898d66/32.png) [@nmarantos](https://community.netdata.cloud/u/nmarantos)\
**Post date:** [October 29, 2021, 2:21pm UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/23 "2021-10-29T14:21:11Z")

</div>

After investigation, i tried to add job in web\_log.conf but from debug mode i see errors to parse csv.

```auto
[DEBUG] main[main] main.go:111 plugin: name=go.d, version=v0.28.2
[DEBUG] main[main] main.go:113 current user: name=netdata, uid=116
[INFO] main[main] agent.go:106 instance is started
[INFO] main[main] setup.go:39 loading config file
[INFO] main[main] setup.go:47 looking for 'go.d.conf' in [/etc/netdata /usr/lib/netdata/conf.d]
[INFO] main[main] setup.go:54 found '/etc/netdata/go.d.conf
[INFO] main[main] setup.go:61 config successfully loaded
[INFO] main[main] agent.go:110 using config: enabled 'true', default_run 'true', max_procs '0'
[INFO] main[main] setup.go:66 loading modules
[INFO] main[main] setup.go:85 enabled/registered modules: 1/59
[INFO] main[main] setup.go:90 building discovery config
[INFO] main[main] setup.go:116 looking for 'web_log.conf' in [/etc/netdata/go.d /usr/lib/netdata/conf.d/go.d]
[INFO] main[main] setup.go:123 found '/etc/netdata/go.d/web_log.conf
[INFO] main[main] setup.go:128 dummy/read/watch paths: 0/1/0
[INFO] discovery[manager] manager.go:90 registered discoverers: [file discovery: [file reader]]
[INFO] discovery[manager] manager.go:95 instance is started
[INFO] discovery[file manager] discovery.go:71 instance is started
[INFO] discovery[file reader] read.go:39 instance is started
[INFO] build[manager] build.go:106 instance is started
[INFO] run[manager] run.go:30 instance is started
[INFO] discovery[file reader] read.go:40 instance is stopped
[DEBUG] build[manager] build.go:153 received config group ('/etc/netdata/go.d/web_log.conf'): 1 jobs (added: 1, removed: 0)
[DEBUG] build[manager] build.go:302 building web_log[nginx_master] job, config: map[__provider__ :file reader __source__ :/etc/netdata/go.d/web_log.conf autodetection_retry:30 log_type:auto module:web_log name:nginx_master path:/var/log/nginx/master.access.log priority:70000 update_every:1]
[DEBUG] web_log[nginx_master] init.go:31 skipping URL patterns creating, no patterns provided
[DEBUG] web_log[nginx_master] init.go:49 skipping custom fields creating, no custom fields provided
[DEBUG] web_log[nginx_master] init.go:74 skipping custom time fields creating, no custom time fields provided
[DEBUG] web_log[nginx_master] init.go:103 starting log reader creating
[DEBUG] web_log[nginx_master] reader.go:71 open log file: /var/log/nginx/master.access.log
[DEBUG] web_log[nginx_master] init.go:108 created log reader, current file '/var/log/nginx/master.access.log'
[DEBUG] web_log[nginx_master] init.go:114 starting parser creating
[DEBUG] web_log[nginx_master] init.go:120 last line: '- - - [29/Oct/2021:16:27:43 +0300] TLSv1.3/TLS_AES_256_GCM_SHA384 "GET /path/to/something HTTP/1.1" 200 1456 "HOST" "https://host_referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36" "-" "-" "W3" rt=0.019 uct="0.012" urt="0.020"'
[DEBUG] web_log[nginx_master] parser.go:75 log_type is auto, will try format auto-detection
[DEBUG] web_log[nginx_master] parser.go:98 starting log type auto-detection
[DEBUG] web_log[nginx_master] parser.go:107 log type is CSV
[DEBUG] web_log[nginx_master] parser.go:112 starting csv log format auto-detection
[DEBUG] web_log[nginx_master] parser.go:113 config: {FieldsPerRecord:-1 Delimiter: TrimLeadingSpace:false Format: CheckField:0xbca280}
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[DEBUG] web_log[nginx_master] parser.go:119 trying format: '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent'
[DEBUG] web_log[nginx_master] parser.go:127 parse: csv parse: parse error on line 1, column 459: bare " in non-quoted-field
[WARN] web_log[nginx_master] weblog.go:112 check failed: create parser: cannot auto-detect log format, use custom log format
[ERROR] web_log[nginx_master] job.go:157 check failed
[DEBUG] web_log[nginx_master] reader.go:127 close log file: /var/log/nginx/master.access.log
[INFO] build[manager] build.go:263 web_log[nginx_master] job detection failed, will retry in 30 seconds

```

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [October 29, 2021, 9:33pm UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/24 "2021-10-29T21:33:24Z")

</div>

> [@nmarantos](#):
>
> i tried to add job in web\_log.conf

Hmm, I was sure you had had it in there 🤔

Yes, the plugin couldn’t detect the log format (tried predefined patterns and failed). Could you show ~~a logline and~~ `log_format` from your nginx.conf?

Oh, I see

```auto
- - - [29/Oct/2021:16:27:43 +0300] TLSv1.3/TLS_AES_256_GCM_SHA384 "GET /path/to/something HTTP/1.1" 200 1456 "HOST" "https://host_referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36" "-" "-" "W3" rt=0.019 uct="0.012" urt="0.020"

```

---

<div class="post-metadata">

**Author:** ![nmarantos](https://avatars.discourse-cdn.com/v4/letter/n/898d66/32.png) [@nmarantos](https://community.netdata.cloud/u/nmarantos)\
**Post date:** [October 30, 2021, 5:35pm UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/25 "2021-10-30T17:35:42Z")

</div>

Yes this is a logline but i change only the http\_referrer and the host for security reasons.  
The following is one of nginx log\_format

```auto
log_format '$http_true_client_ip - $remote_user [$time_local] '
                '$ssl_protocol/$ssl_cipher '
                '"$request" $status $body_bytes_sent '
                '"$host" "$http_referer" "$http_user_agent" "$http_x_request_id" "$correlationId" "$sent_http_prx" '
                'rt=$request_time uct="$upstream_connect_time" urt="$upstream_response_time"';

```

I was wondering why in a server with nginx opensource and netdata is working properly.  
To be more specific, in the other server with opensource nginx it works with python plugin web log , but now i tried with both plugins and i see the same result.

Is there any issue with web\_log config in plugins and nginx plus?  
Is there any idea or other configuration (custom format for the specific?) for this setup?

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [October 31, 2021, 11:39am UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/26 "2021-10-31T11:39:25Z")

</div>

@nmarantos There is a list of predefined patterns (works for nginx/apache), here it is (go version, but the same logic applies for py): [https://github.com/netdata/go.d.plugin/blob/83f00839d2675d308dddc0a5f2cf02e999cbe06f/modules/weblog/parser.go#L36-L45](https://github.com/netdata/go.d.plugin/blob/83f00839d2675d308dddc0a5f2cf02e999cbe06f/modules/weblog/parser.go#L36-L45)

The plugin tries all of them and if none works you get

> [WARN] web\_log[nginx\_master] weblog.go:112 check failed: create parser: cannot auto-detect log format, use custom log format

Is the `log_format` you shared default? Or did you change it?

---

<div class="post-metadata">

**Author:** ![nmarantos](https://avatars.discourse-cdn.com/v4/letter/n/898d66/32.png) [@nmarantos](https://community.netdata.cloud/u/nmarantos)\
**Post date:** [November 1, 2021, 9:43am UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/27 "2021-11-01T09:43:12Z")

</div>

Ηι @ilyam8 ,

```auto
$http_true_client_ip $ssl_protocol/$ssl_cipher "$http_referer" "$http_user_agent" "$http_x_request_id" "$upstreamCorrelationId" "$sent_http_prx"

```

These were added in the log format which i use, but i had not any issue in previous server with nginx opensource and python.d plugin.

**UPDATE**

I changed log format like the last predefined and it works.  
How can i add the above fields in the log format i want?

The log format with which it played and i want fields to be added to it.

```auto
$host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time

```

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [November 2, 2021, 8:52am UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/28 "2021-11-02T08:52:12Z")

</div>

There is a [list of known fields](https://github.com/netdata/go.d.plugin/tree/master/modules/weblog#known-fields). All of them can be used when creating a [custom pattern](https://github.com/netdata/go.d.plugin/tree/master/modules/weblog#custom-log-format).

A note: a space is a default separator when using CSV format. So

> [@nmarantos](#):
>
> `$ssl_protocol/$ssl_cipher`

should be `$ssl_protocol $ssl_cipher` (if we are interested in SSL metrics/charts).

* * *

Python version uses regular expressions (see the [list of predefined regular expressions](https://github.com/netdata/netdata/blob/95a2f16795be38b26bcddbde24e4d679dfef4427/collectors/python.d.plugin/web_log/web_log.chart.py#L627-L669)).

Go version [supports the following parsers](https://github.com/netdata/go.d.plugin/tree/master/modules/weblog#log-parsers):

- CSV (default)
- JSON
- RegExp (python uses only this)
- LTSV

* * *

> [@nmarantos](#):
>
> ```auto
> log_format '$http_true_client_ip - $remote_user [$time_local] '
> '$ssl_protocol/$ssl_cipher '
> '"$request" $status $body_bytes_sent '
> '"$host" "$http_referer" "$http_user_agent" "$http_x_request_id" "$correlationId" "$sent_http_prx" '
> 'rt=$request_time uct="$upstream_connect_time" urt="$upstream_response_time"';
> 
> ```

So you are going to use this format, correct?

> [@nmarantos](#):
>
> These were added in the log format which i use

Do you remember what was the default Nginx plus format?

---

<div class="post-metadata">

**Author:** ![nmarantos](https://avatars.discourse-cdn.com/v4/letter/n/898d66/32.png) [@nmarantos](https://community.netdata.cloud/u/nmarantos)\
**Post date:** [November 8, 2021, 9:17am UTC](https://community.netdata.cloud/t/web-log-conf-for-nginx/39/29 "2021-11-08T09:17:54Z")

</div>

Hi @ilyam8,

From nginx.conf i changed my log format to one line and in web\_log.conf in custom csv\_config i configured it with the same raw and the same type. Maybe before some variable did not recognize it or was not in the same order.  
Now with this way, it is working.

[Previous page](https://community.netdata.cloud/t/web-log-conf-for-nginx/39.md?page=1)