Prometheus endpoints on Netdata agent

General
1

Hello. I am new to Netdata and I am attempting to configure the following on a K8 stack:

  • Using Python prometheus-client, I created a Counter metric and exposed this on port 8000 with start_http_server(8000) in my python code running on a pod
  • Modified prometheus configmap to add a new target to scrape port 8000 and restarted prometheus server.
  • New counter metric is now visible in Prometheus
  • After installing Netdata and launching agent, I would like to view this new Prometheus target/metric on Netdata agent.

Does the Netdata Prometheus collector automatically detect this new target? I am having no luck viewing the metric on Netdata agent. I am thinking that a config change is needed but am having no luck so far. I added ‘127.0.0.1:8000/metrics’ to prometheus.conf on the parent pod, but still no luck. I also used ‘pod-ip:8000/metrics’ in prometheus.conf…no luck.

Any input much appreciated!

2

Hi, @rgary1964. I believe it should work if:

  • Netdata installed inside the K8s cluster (not directly on worker nodes).
  • The pod (the one with 8000) has prometheus.io/scrape: true label.
  • 8000 is exposed in the container of the Pod.
3

Hi @ilyam8. Thank you for the info! I will give this a try.

4

@ilyam8 this is great, but where in netdata cloud can I view / search for the prometheus metrics? Thanks.

5

Nevermind @ilyam8, I just saw them there :slight_smile: