dbengine growing above configured limits (1.46.2)

Help
, ,
1

Version: netdata v1.46.2
OS: debian 11

preliminary note

The “config webpage” http://host.domain:19999/netdata.conf is showing the currently active configuration. That’s of great help.
If I change /etc/netdata/netdata.conf and then systemctl restart netdata I can observe that the “…:19999/netdata.conf” page is changing accordingly.

Environment/Browser/Agent’s version etc

Version: netdata v1.46.2
OS: debian 11

Problem/Question 01

It seems not possible to set the disk space of “Tier 0” below “256”.

If I set dbengine tier 0 disk space MB = 257" in the netdata.conf and restart, it is represented like that on the config webpage - it will show: “dbengine tier 0 disk space MB = 257”

If I set “dbengine tier 0 disk space MB = 255” in the netdata.conf, then on the config webpage it changes to “dbengine tier 0 disk space MB = 256”

If I set dbengine tier 0 disk space MB = 100 in the netdata.conf, then on the config webpage it changes to dbengine tier 0 disk space MB = 256

Problem/Question 02

The configuration of retention for each “tier” seems to not have effect.

With below printed configuration (see 1) and it’s loaded representation (see 2), the size of dbengine directories still grows ofter the MB-size limits:

2024-07-31

$ sudo du -sh /var/cache/netdata/dbengine*
47M     /var/cache/netdata/dbengine-tier1   # already over the limit
216M    /var/cache/netdata/dbengine            # still have to wait to check
                                                                                   # dbengine-tier2 still has to grow

$ sudo du -bh /var/cache/netdata/dbengine*
52M     /var/cache/netdata/dbengine-tier1
232M    /var/cache/netdata/dbengine

$ sudo systemctl status netdata
netdata.service - Real time performance monitoring
   Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2024-07-29 15:13:51 CEST; 1 day 22h ago

2024-08-01 (seems the service restarted in the morning)

$ sudo du -sh /var/cache/netdata/dbengine*
67M     /var/cache/netdata/dbengine-tier1   # over the limit
305M    /var/cache/netdata/dbengine            # over the limit
5.1M    /var/cache/netdata/dbengine-tier2   # dbengine-tier2 still has to grow

$ sudo du -bh /var/cache/netdata/dbengine*
76M     /var/cache/netdata/dbengine-tier1
327M    /var/cache/netdata/dbengine
5.7M    /var/cache/netdata/dbengine-tier2

$ sudo systemctl status netdata
netdata.service - Real time performance monitoring
   Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2024-08-01 00:16:49 CEST; 9h ago

Problem/Question 03

When I would set the limits to a lower value and restart the service afterwards, e.g.:

dbengine tier 1 disk space MB = 5
dbengine tier 2 disk space MB = 2

Question: What should happen?

What I observe: the dbengine directories are growing even further
What I would expect: that there is some kind of “purging”, so that the directories shrink below the defined threshold (at least it was kinda like that behavior with the old/legacy engine - after a restart it would shrink below the threshold and would grow again…)

Configuration

(1) My local configuration:

[global]
bind socket to IP = *
run as user = netdata

[health]
enabled = no

[db]
mode = dbengine
storage tiers = 3

#Tier 0, per second data. Set to 0 for no limit.
#dbengine tier 0 disk space MB = 257            # works
dbengine tier 0 disk space MB = 100             # does not work, will set "dbengine tier 0 disk space MB = 256"
dbengine tier 0 retention days = 14

# Tier 1, per minute data. Set to 0 for no limit.
dbengine tier 1 disk space MB = 45
dbengine tier 1 retention days = 90

# Tier 2, per hour data. Set to 0 for no limit.
dbengine tier 2 disk space MB = 25
dbengine tier 2 retention days = 730

[ml]
enabled = no

(2) Web representation (stripped of comments):

[global]
  run as user = netdata

[db]
  mode = dbengine
  storage tiers = 3
  dbengine tier 0 disk space MB = 256
  dbengine tier 0 retention days = 14
  dbengine tier 1 disk space MB = 45
  dbengine tier 1 retention days = 90
  dbengine tier 2 disk space MB = 25
  dbengine tier 2 retention days = 730

[ml]
  enabled = no

[health]
  enabled = no

[web]
  bind to = *

What I expected to happen

If I set dbengine tier 0 disk space MB = 100 in the /etc/netdata/netdata.conf, I expect config webpage to changes to dbengine tier 0 disk space MB = 100

If I set a limit, the directory should not grow over that limit

Relevant docs you followed/actions you took to solve the issue

I looked into following documentations for explanation:

I looked into following threads for eventually alike topics, but it didn’t solve my problems:

image

2

Answered on GitHub.

1 Like
3

just adding for future visitors:

this would be a correct db-config inside netdata.conf:

[db]
mode = dbengine
storage tiers = 3
dbengine tier 0 disk space MB = 256
dbengine tier 1 disk space MB = 256
dbengine tier 2 disk space MB = 256

No “multihost”!! In above context following would be wrong: dbengine tier 0 multihost disk space MB = 256

Answer question 1/2)
256 is the minimal size (a value below would still resolve to “256” in the web-interpretation in https://server01.xyz:19999/netdata.conf )

Answer question 3)
If the size of the “dbengine tier”-folder has grown to 700MB and one would configure the limit like dbengine tier 0 disk space MB = 256 and resetart with e.g. systemctl restart netdata the size of the respective dbengine-folder would directly shrink below that value