Stream child metric to parent in Kubernetes

Help
, ,
1

Hi guys!

I was wondering if anybody has experience with a setup like this:

  1. Parent node is running in Kubernetes cluster, behind Ingress, let’s call it netdata.k8s.local
  2. Ingress exposes port 443, so netdata dashboard is accessible over https://netdata.k8s.local`
  3. Child node, that is running outside of Kubernetes(instance-01)

I try to stream child metrics outside of Kubernetes to the parent node inside Kubernetes, through Ingress, but I have error like this:

2020-09-10 23:21:14: netdata ERROR : STREAM_SENDER[Sellars] : STREAM Sellars [send to tcp:netdata.k8s.local:443]: server is not replying properly (is it a netdata?).

If I change child configuration to stream to normal server, not in Kubernetes - it works.
Child nodes inside Kubernetes cluster can stream to parent inside Kubernetes as well.

Is there a way to make Kubernetes streaming work?
Can it be an issue with certificates in Ingress? (I use let’s encrypt)

Any tips would be appreciated :slight_smile:

2

Thanks @Rybue for sharing this information. It sure helps a lot of our users, as kubernetes is super popular amongst netdata users.

@joel @ilyam8 perhaps this content is interesting to you for our docs/readmes.

Cheers :slight_smile:

3

Ok, so after playing around with Ingress, it seems to be not possible to just use normal port 80/443.
So, I used workaround like this:

  1. Add configmap to Ingress to expose additional TCP port and updated Ingress service to open that port(Exposing TCP and UDP services - NGINX Ingress Controller), for example 19999
  2. This port will point directly to Netdata parent service, so mapping looks like 19999 on Ingress(LB) to 19999 on netdata service
  3. Use that 19999 port and Ingress DNS name in child configuration.

Hopefully it will be usefull for somebody.

1 Like