Monitoring Ingress HTTP traffic from all namespaces.

I found the following item on the Netdata documentation page.
https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/httpcheck/

After that I tried to list this plugin in my Values.yaml and start a helm upgrade.

helm upgrade -f values.yaml netdata netdata/netdata,
             --namespace monitoring,
             --set parent.nodeSelector."beta\.kubernetes\.io/os"=linux,
             --set child.nodeSelector."beta\.kubernetes\.io/os"=linux,
             --set parent.database.volumesize=4Gi,
             --set-file sd.child.configmap.from.value=./child.yml

Unfortunately without success. Do you have any suggestions how to achieve this goal ?

Eventuel also interesting to know. My service discovery looks like this.

Possibly also interesting to know. My service discovery looks like this.

> kubectl -n monitoring logs netdata-child-5zfjs -c sd

{"level":"info","component":"pipeline manager","time":"2021-06-15 22:18:54","message":"instance is started"}
{"level":"info","component":"k8s config provider","time":"2021-06-15 22:18:54","message":"instance is started"}
{"level":"info","component":"export manager","time":"2021-06-15 22:18:55","message":"registered: '[file exporter (/export/go.d.yml)]'"}
{"level":"info","component":"discovery manager","time":"2021-06-15 22:18:55","message":"registered: [k8s discovery manager]"}
{"level":"info","component":"pipeline manager","time":"2021-06-15 22:18:55","message":"received a new config, starting a new pipeline ('k8s/cmap/monitoring/netdata-child-sd-config-map:config.yml')"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"export manager","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"discovery manager","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"file export","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"k8s discovery manager","time":"2021-06-15 22:18:55","message":"registered: [k8s pod discovery]"}
{"level":"info","component":"k8s pod discovery","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"received '37' group(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/tenant-service-747c4f5577-9qdd9' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/event-service-66c6d8f5fd-dgmm7' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/operations-center-545c97df4c-5lxdf' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/inventory-service-74c66fc788-5grrj' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/event-service-76874db675-sfdbt' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/azure-cni-networkmonitor-pwbnc' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/aux-service-6f8869d8c7-mqwnk' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/nginx-ingress-stg-ingress-nginx-controller-64fb8c84f8-fkpf7' with 5 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/mqtt-69d8ff6cb7-hm5fz' with 3 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/mqtt-5fdb4b8599-5cxk9' with 0 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/config-service-7cdbc9c5b4-kkjtv' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/gatekeeper-system/gatekeeper-audit-6dcf64cf84-ll62h' with 2 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/event-service-84bbdc796c-v7sfc' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/kube-proxy-szn2z' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/gatekeeper-system/gatekeeper-controller-6dd6b7f4c9-5swjf' with 3 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/aux-service-655b8bd78f-bqq7m' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/config-service-7fbb46876d-rmwtt' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/config-service-554578664b-r6r4z' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/mqtt-6fd7796798-rzjnw' with 3 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/tenant-service-79469ddb56-zm426' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/coredns-9d6c6c99b-79rnl' with 3 target(s)"}
{"level":"info","component":"build manager","time":"2021-06-15 22:19:00","message":"built 1 config(s) for target 'kube-system_coredns-9d6c6c99b-79rnl_coredns_tcp_9153'"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"group 'k8s/pod/kube-system/coredns-9d6c6c99b-79rnl': new/stale config(s) 1/0"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/operations-center-7f69589fd7-6k9qp' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/azure-ip-masq-agent-bdrgd' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/monitoring/netdata-child-5zfjs' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/azure-policy-798f4cbf69-868hf' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/operations-center-6657f65596-zh6sb' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/nginx-ingress-dev-ingress-nginx-controller-5b665f8d9c-h8j8m' with 5 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/identity-service-6cbcb4665f-mbxhl' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/inventory-service-66d6bc89b-5v5nz' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/internal/dev-service-7b7b6d4bf9-nnrlb' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/identity-service-85644d4f95-475qk' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/config-service-554578664b-8bjks' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/nginx-ingress-qa-ingress-nginx-controller-6dfb6b55c4-f6nk4' with 5 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/inventory-service-64bf4bd878-2696m' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/operations-center-7f69589fd7-cznff' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/identity-service-899796b88-992pn' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/omsagent-fn2tc' with 2 target(s)"}
{"level":"info","component":"file export","time":"2021-06-15 22:19:01","message":"wrote 1 config(s) to '/export/go.d.yml'"}

Thanks in advance for any help!

The configuration looks alright to me.

Can you please send us the error.log that you should find in /var/log/netdata of the child pod that runs Netdata?

Do you have any input @ilyam8 regarding the k8s setup?

Thanks for approving my account and for the quick feedback.

Logs from kubectl -n monitoring logs netdata-child-zkzl7 -c netdata

ZeroBin.net

Logs very to many chars

This passage in the logs irritates me

2021-06-16 11:51:22: python.d ERROR: httpcheck[httpcheck] : URL is not defined or type is not
2021-06-16 11:51:22: python.d INFO: plugin[main] : httpcheck[httpcheck] : check failed

My Child Config

child:
  claiming:
    enabled: true
    token: "xxxxxx"
    rooms: "xxxxxxx"
  
  nodeSelector:
    beta.kubernetes.io/os: linux  
  
  configs:
    netdata:
      data: |
        [global]
          memory mode = ram
          history = 21600
        [health]
          enabled = no
    httpcheck:
      path: /usr/lib/netdata/conf.d/go.d/httpcheck.conf
      data: |
        [global]
          update_every        : 1
          autodetection_retry : 0
          priority            : 70000

        [jobs]
          jobs:
            - name: devingress
              url: https://dev.mydomain.cloud
              status_accepted: [200, 300, 400]
              response_match: <title>dev ingress<\/title>
              timeout: 5

            - name: stgingress
              url: https://stg.mydomain.cloud
              status_accepted: [200, 300, 400]
              response_match: <title>stg ingress<\/title>
              timeout: 5

it’s not like i don’t see any info in netdata.
But not the one I need. Effectively I only need the HTTP requests that go through the ingress

If more information is needed I am at your disposal.

Hi @chefcook, sorry for the delay in getting back to you.

Gave a short answer in Monitoring Ingress HTTP traffic from all namespaces. · Issue #221 · netdata/helmchart · GitHub

TL;DR

Unfortunately, it is not possible atm. Fortunately, we plan to have that fixed this quarter :smiley:


If you have any other questions or wants me to provide any details - I’m happy to answer.