# Prometheus metric grouping broken?

**URL:** <https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699>\
**Category:** Help\
**Tags:** configuration, agent, collectors\
**Created:** [December 31, 2022, 5:14am UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699 "2022-12-31T05:14:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![crb](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/crb/32/2500_2.png) [@crb](https://community.netdata.cloud/u/crb)\
**Post date:** [December 31, 2022, 5:14am UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699/1 "2022-12-31T05:14:21Z")

</div>

## Problem/Question

I’ve configured netdata to monitor a prometheus endpoint with a large number of counters that I’d like to group. Either my config is wrong (likely) or the grouping configuration as described in the prometheus module docs no longer seems to work.

To me it looks like the grouping functionality was removed in this PR:  
[https://github.com/netdata/go.d.plugin/pull/1004](https://github.com/netdata/go.d.plugin/pull/1004)

## Relevant docs you followed/actions you took to solve the issue

[README describing grouping config](https://github.com/netdata/go.d.plugin/blob/master/modules/prometheus/README.md#time-series-grouping)

## Environment/Browser/Agent’s version etc

Running v1.37.0-89-nightly in a container

## What I expected to happen

The prometheus endpoint returns data like the following:

```auto
unifipoller_device_cpu_utilization_ratio{name="U6-Lite",site_name="Default (default)",type="uap"} 0.042
unifipoller_device_cpu_utilization_ratio{name="UAP-AC-Pro",site_name="Default (default)",type="uap"} 0.085
unifipoller_device_cpu_utilization_ratio{name="US-8",site_name="Default (default)",type="usw"} 0.563
unifipoller_device_cpu_utilization_ratio{name="USG-Pro-4",site_name="Default (default)",type="ugw"} 0.07
unifipoller_device_cpu_utilization_ratio{name="USW-Lite-16-PoE",site_name="Default (default)",type="usw"} 0.10400000000000001

```

The module config is as follows:

```auto
jobs:                                                                                              
  - name: unifi_exporter                                                                                      
    url: 'http://localhost:9031/metrics'                                                               
    selector:                                                                                                           
      allow:                                                                                                  
        - unifipoller_device_cpu_utilization_ratio
    group:                                                                                                    
      - selector: unifipoller_device_cpu_utilization_ratio                                                                           
        by_label: site_name

```

What I expect/want is one chart with 5 lines (1 per “name”). What I get is 5 separate charts. Thanks for any help with my config or misunderstanding!

---

<div class="post-metadata">

**Author:** ![andrewm4894](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/andrewm4894/32/125_2.png) [@andrewm4894](https://community.netdata.cloud/u/andrewm4894)\
**Post date:** [January 6, 2023, 5:12pm UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699/2 "2023-01-06T17:12:49Z")

</div>

hi @crb welcome! apologies for delay here. i will read this and see if i can help you.

---

<div class="post-metadata">

**Author:** ![andrewm4894](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/andrewm4894/32/125_2.png) [@andrewm4894](https://community.netdata.cloud/u/andrewm4894)\
**Post date:** [January 6, 2023, 5:22pm UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699/3 "2023-01-06T17:22:36Z")

</div>

@crb are you looking at the chart from within Netdata Cloud or just the agent dashboard?

just reading the docs myself and i think your config looks correct as best i can tell.

@ilyam8 does the config above look correct to you?

---

<div class="post-metadata">

**Author:** ![andrewm4894](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/andrewm4894/32/125_2.png) [@andrewm4894](https://community.netdata.cloud/u/andrewm4894)\
**Post date:** [January 6, 2023, 5:48pm UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699/4 "2023-01-06T17:48:11Z")

</div>

I just tried to reproduce this with a collection from the demo node like below focusing on these metrics:

```auto
# HELP node_filesystem_files Filesystem total file nodes.
# TYPE node_filesystem_files gauge
node_filesystem_files{device="/dev/vda1",fstype="ext4",mountpoint="/"} 3.2256e+06
node_filesystem_files{device="/dev/vda15",fstype="vfat",mountpoint="/boot/efi"} 0
node_filesystem_files{device="lxcfs",fstype="fuse.lxcfs",mountpoint="/var/lib/lxcfs"} 0
node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 126146
node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run/lock"} 126146

```

So if i group by `fstype` i’d expect 4.

```auto
jobs:

  - name: node_exporter_demo_fstype
    url: 'https://node.demo.do.prometheus.io/metrics'
    selector:
      allow:
        - node_filesystem_files
    group:
      - selector: node_filesystem_files
        by_label: fstype

```

In the charts endpoint of netdata i see 5 not 4 (so in line with same issue you have @crb )

@ilyam8 would i be right in expecting these two to be one chart? eg i want netdata to ignore the `device` and `mountpoint` labels.

```auto
prometheus_node_exporter_demo_fstype.node_filesystem_files-device=tmpfs-fstype=tmpfs-mountpoint=/run
prometheus_node_exporter_demo_fstype.node_filesystem_files-device=tmpfs-fstype=tmpfs-mountpoint=/run/lock

```

```auto
"prometheus_node_exporter_demo_fstype.node_filesystem_files-device=/dev/vda1-fstype=ext4-mountpoint=/": {
			"id": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device=/dev/vda1-fstype=ext4-mountpoint=/",
			"name": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device __dev_vda1-fstype_ext4-mountpoint__",
			"type": "prometheus_node_exporter_demo_fstype",
			"family": "node_filesystem",
			"context": "prometheus.node_exporter_demo_fstype.node_filesystem_files",
			"title": "Filesystem total file nodes (prometheus_node_exporter_demo_fstype.node_filesystem_files-device __dev_vda1-fstype_ext4-mountpoint__ )",
			"priority": 70000,
			"plugin": "go.d",
			"module": "prometheus",
			"units": "files",
			"data_url": "/api/v1/data?chart=prometheus_node_exporter_demo_fstype.node_filesystem_files-device __dev_vda1-fstype_ext4-mountpoint__",
			"chart_type": "line",
			"duration": 10,
			"first_entry": 0,
			"last_entry": 0,
			"update_every": 10,
			"dimensions": {
				"node_filesystem_files": { "name": "node_filesystem_files" }
			},
			"chart_variables": {
			},
			"green": null,
			"red": null,
			"alarms": {

			},
			"chart_labels": {
				"_collect_plugin":"go.d",
				"_collect_module":"prometheus",
				"device":"/dev/vda1",
				"fstype":"ext4",
				"mountpoint":"/",
				"_collect_job":"node_exporter_demo_fstype"
			},
			"functions": {
			}
		},
		"prometheus_node_exporter_demo_fstype.node_filesystem_files-device=/dev/vda15-fstype=vfat-mountpoint=/boot/efi": {
			"id": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device=/dev/vda15-fstype=vfat-mountpoint=/boot/efi",
			"name": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device __dev_vda15-fstype_vfat-mountpoint__ boot_efi",
			"type": "prometheus_node_exporter_demo_fstype",
			"family": "node_filesystem",
			"context": "prometheus.node_exporter_demo_fstype.node_filesystem_files",
			"title": "Filesystem total file nodes (prometheus_node_exporter_demo_fstype.node_filesystem_files-device __dev_vda15-fstype_vfat-mountpoint__ boot_efi)",
			"priority": 70000,
			"plugin": "go.d",
			"module": "prometheus",
			"units": "files",
			"data_url": "/api/v1/data?chart=prometheus_node_exporter_demo_fstype.node_filesystem_files-device __dev_vda15-fstype_vfat-mountpoint__ boot_efi",
			"chart_type": "line",
			"duration": 10,
			"first_entry": 0,
			"last_entry": 0,
			"update_every": 10,
			"dimensions": {
				"node_filesystem_files": { "name": "node_filesystem_files" }
			},
			"chart_variables": {
			},
			"green": null,
			"red": null,
			"alarms": {

			},
			"chart_labels": {
				"_collect_plugin":"go.d",
				"_collect_module":"prometheus",
				"device":"/dev/vda15",
				"fstype":"vfat",
				"mountpoint":"/boot/efi",
				"_collect_job":"node_exporter_demo_fstype"
			},
			"functions": {
			}
		},
		"prometheus_node_exporter_demo_fstype.node_filesystem_files-device=lxcfs-fstype=fuse.lxcfs-mountpoint=/var/lib/lxcfs": {
			"id": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device=lxcfs-fstype=fuse.lxcfs-mountpoint=/var/lib/lxcfs",
			"name": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device_lxcfs-fstype_fuse.lxcfs-mountpoint__var_lib_lxcfs",
			"type": "prometheus_node_exporter_demo_fstype",
			"family": "node_filesystem",
			"context": "prometheus.node_exporter_demo_fstype.node_filesystem_files",
			"title": "Filesystem total file nodes (prometheus_node_exporter_demo_fstype.node_filesystem_files-device_lxcfs-fstype_fuse.lxcfs-mountpoint__var_lib_lxcfs)",
			"priority": 70000,
			"plugin": "go.d",
			"module": "prometheus",
			"units": "files",
			"data_url": "/api/v1/data?chart=prometheus_node_exporter_demo_fstype.node_filesystem_files-device_lxcfs-fstype_fuse.lxcfs-mountpoint__var_lib_lxcfs",
			"chart_type": "line",
			"duration": 10,
			"first_entry": 0,
			"last_entry": 0,
			"update_every": 10,
			"dimensions": {
				"node_filesystem_files": { "name": "node_filesystem_files" }
			},
			"chart_variables": {
			},
			"green": null,
			"red": null,
			"alarms": {

			},
			"chart_labels": {
				"_collect_plugin":"go.d",
				"_collect_module":"prometheus",
				"device":"lxcfs",
				"fstype":"fuse.lxcfs",
				"mountpoint":"/var/lib/lxcfs",
				"_collect_job":"node_exporter_demo_fstype"
			},
			"functions": {
			}
		},
		"prometheus_node_exporter_demo_fstype.node_filesystem_files-device=tmpfs-fstype=tmpfs-mountpoint=/run": {
			"id": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device=tmpfs-fstype=tmpfs-mountpoint=/run",
			"name": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device_tmpfs-fstype_tmpfs-mountpoint__run",
			"type": "prometheus_node_exporter_demo_fstype",
			"family": "node_filesystem",
			"context": "prometheus.node_exporter_demo_fstype.node_filesystem_files",
			"title": "Filesystem total file nodes (prometheus_node_exporter_demo_fstype.node_filesystem_files-device_tmpfs-fstype_tmpfs-mountpoint__run)",
			"priority": 70000,
			"plugin": "go.d",
			"module": "prometheus",
			"units": "files",
			"data_url": "/api/v1/data?chart=prometheus_node_exporter_demo_fstype.node_filesystem_files-device_tmpfs-fstype_tmpfs-mountpoint__run",
			"chart_type": "line",
			"duration": 10,
			"first_entry": 0,
			"last_entry": 0,
			"update_every": 10,
			"dimensions": {
				"node_filesystem_files": { "name": "node_filesystem_files" }
			},
			"chart_variables": {
			},
			"green": null,
			"red": null,
			"alarms": {

			},
			"chart_labels": {
				"_collect_plugin":"go.d",
				"_collect_module":"prometheus",
				"device":"tmpfs",
				"fstype":"tmpfs",
				"mountpoint":"/run",
				"_collect_job":"node_exporter_demo_fstype"
			},
			"functions": {
			}
		},
		"prometheus_node_exporter_demo_fstype.node_filesystem_files-device=tmpfs-fstype=tmpfs-mountpoint=/run/lock": {
			"id": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device=tmpfs-fstype=tmpfs-mountpoint=/run/lock",
			"name": "prometheus_node_exporter_demo_fstype.node_filesystem_files-device_tmpfs-fstype_tmpfs-mountpoint__run_lock",
			"type": "prometheus_node_exporter_demo_fstype",
			"family": "node_filesystem",
			"context": "prometheus.node_exporter_demo_fstype.node_filesystem_files",
			"title": "Filesystem total file nodes (prometheus_node_exporter_demo_fstype.node_filesystem_files-device_tmpfs-fstype_tmpfs-mountpoint__run_lock)",
			"priority": 70000,
			"plugin": "go.d",
			"module": "prometheus",
			"units": "files",
			"data_url": "/api/v1/data?chart=prometheus_node_exporter_demo_fstype.node_filesystem_files-device_tmpfs-fstype_tmpfs-mountpoint__run_lock",
			"chart_type": "line",
			"duration": 10,
			"first_entry": 0,
			"last_entry": 0,
			"update_every": 10,
			"dimensions": {
				"node_filesystem_files": { "name": "node_filesystem_files" }
			},
			"chart_variables": {
			},
			"green": null,
			"red": null,
			"alarms": {

			},
			"chart_labels": {
				"_collect_plugin":"go.d",
				"_collect_module":"prometheus",
				"device":"tmpfs",
				"fstype":"tmpfs",
				"mountpoint":"/run/lock",
				"_collect_job":"node_exporter_demo_fstype"
			},
			"functions": {
			}
		},

```

---

<div class="post-metadata">

**Author:** ![ilyam8](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/ilyam8/32/134_2.png) [@ilyam8](https://community.netdata.cloud/u/ilyam8)\
**Post date:** [January 6, 2023, 5:48pm UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699/5 "2023-01-06T17:48:14Z")

</div>

Yes, the grouping as it was - removed. Because it was the wrong implementation (as it turned out).

> What I expect/want is one chart with 5 lines (1 per “name”).

That is possible when using Cloud UI (go to the chart, change “group by” to “name”). The local dashboard doesn’t support this at the moment.

---

<div class="post-metadata">

**Author:** ![andrewm4894](https://yyz1.discourse-cdn.com/flex029/user_avatar/community.netdata.cloud/andrewm4894/32/125_2.png) [@andrewm4894](https://community.netdata.cloud/u/andrewm4894)\
**Post date:** [January 6, 2023, 5:53pm UTC](https://community.netdata.cloud/t/prometheus-metric-grouping-broken/3699/6 "2023-01-06T17:53:18Z")

</div>

oh yep - that solves my reproduction too - good to know

 ![image](https://canada1.discourse-cdn.com/flex029/uploads/netdata2/original/2X/8/8b568a123bdef3e142dffa73cbcc90a943032875.png)