How to troubleshoot poor performance in the WebUI

General
1

Hi everyone,

Quick question. How can we troubleshoot slow chart loads / performance issues in the WebUI? For instance, quite often when the webUI loads it will either take a while for the page to initially render or if we scroll to a specific chart down the page that chart takes a while to render.

Here’s a screenshot of the developer tools:

Screen Shot 2021-12-03 at 7.47.09 AM
Screen Shot 2021-12-03 at 7.47.09 AM3220×260 79 KB

If I do the comparable cURL command with timing enabled:

time curl 'http://vault-dbsa102:19999/api/v1/data?chart=vault.fetch_times&_=1638542499749&format=json&points=382&group=average&gtime=0&options=ms%7Cflip%7Cjsonwrap%7Cnonzero&after=-420' \
  -H 'Connection: keep-alive' \
  -H 'Pragma: no-cache' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Cache-Control: no-cache, no-store' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36' \
  -H 'Sec-GPC: 1' \
  -H 'Referer: http://vault-dbsa102:19999/' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H 'Cookie: ph_mqkwGT0JNFqO-zX2t0mW6Tec9yooaVu7xCBlXtHnt5Y_posthog=%7B%22distinct_id%22%3A%2217d480d8ba9930-074bb2e7cc14f4-1e396452-1fa400-17d480d8baaaa9%22%2C%22%24device_id%22%3A%2217d480d8ba9930-074bb2e7cc14f4-1e396452-1fa400-17d480d8baaaa9%22%7D' \
  --compressed

I see that it takes 16 seconds

0.01s user 0.01s system 0% cpu 16.031 total

Here’s the global/web config

$ curl -s http://localhost:19999/netdata.conf
# netdata configuration
#
# You can download the latest version of this file, using:
#
#  wget -O /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
# or
#  curl -o /etc/netdata/netdata.conf http://localhost:19999/netdata.conf
#
# You can uncomment and change any of the options below.
# The value shown in the commented settings, is the default value.
#

# global netdata configuration

[global]
	config directory = /etc/netdata/conf.d
	history = 43932
	page cache size = 64
	dbengine multihost disk space = 512
	memory mode = save
	errors to trigger flood protection = 1000
	update every = 5
	# run as user = netdata
	# glibc malloc arena max for plugins = 1
	# glibc malloc arena max for netdata = 1
	# hostname = vault-dbsa102.prod.ostk.com
	# stock config directory = /usr/lib64/netdata/conf.d
	# log directory = /var/log/netdata
	# web files directory = /usr/share/netdata/web
	# cache directory = /var/cache/netdata
	# lib directory = /var/lib/netdata
	# home directory = /var/log/netdata
	# lock directory = /var/lib/netdata/lock
	# plugins directory = "/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"
	# dbengine disk space = 256
	# host access prefix =
	# memory deduplication (ksm) = yes
	# TZ environment variable = :/etc/localtime
	# timezone = America/Denver
	# debug flags = 0x0000000000000000
	# debug log = /var/log/netdata/debug.log
	# error log = /var/log/netdata/error.log
	# access log = /var/log/netdata/access.log
	# facility log = daemon
	# errors flood protection period = 1200
	# OOM score = 1000
	# process scheduling policy = idle
	# pthread stack size = 8388608
	# cleanup obsolete charts after seconds = 3600
	# gap when lost iterations above = 1
	# cleanup orphan hosts after seconds = 3600
	# delete obsolete charts files = yes
	# delete orphan hosts files = yes
	# enable zero metrics = no
	# dbengine extent pages = 64

[web]
	web files owner = netdata
	web files group = netdata
	bind to = 0.0.0.0
	allow management from = localhost dbaautomation*
	# ssl key = /etc/netdata/conf.d/ssl/key.pem
	# ssl certificate = /etc/netdata/conf.d/ssl/cert.pem
	# tls version = 1.3
	# tls ciphers = none
	# ses max window = 15
	# des max window = 15
	# mode = static-threaded
	# listen backlog = 4096
	# default port = 19999
	# disconnect idle clients after seconds = 60
	# timeout for first request = 60
	# accept a streaming request every seconds = 0
	# respect do not track policy = no
	# x-frame-options response header =
	# allow connections from = localhost *
	# allow connections by dns = heuristic
	# allow dashboard from = localhost *
	# allow dashboard by dns = heuristic
	# allow badges from = *
	# allow badges by dns = heuristic
	# allow streaming from = *
	# allow streaming by dns = heuristic
	# allow netdata.conf from = localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.* 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.* 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.* 172.31.* UNKNOWN
	# allow netdata.conf by dns = no
	# allow management by dns = heuristic
	# enable gzip compression = yes
	# gzip compression strategy = default
	# gzip compression level = 3
	# web server threads = 1
	# web server max sockets = 256
	# custom dashboard_info.js =

Netdata version

netdata -v
netdata v1.31.0
2

Hi, @mjtice.

Can you try the same command but on localhost? So we can exclude network problems.

3

In addition, go the “Netdata” section (from the right navbar or scroll to the bottom) for the charts that show the Netdata web server’s request rate and latency. It may be serving a lot more requests than you think.

Such requests are normally handled in a few milliseconds, so there’s definitely something bad happening on your node.

4

So… I’m not sure why I didn’t bother checking from the localhost or even within the same data center but it looks like this slowness is related to my VPN connection. I had a coworker validate with the same result.

I’ll see if my networking team can investigate this. Thanks for your help, guys!