HOWTO - HTTP EndPoint Collector with Cookie and user/pass

Hi Everyone,

I got some requests and I need your help. We would like to add a few http endpoint collectors but in order to get the results I have first to pass the proper login. This is a 3 step process and from bash I can do that as below example:

curl '' \
  -H 'Referer:' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0' \
  --insecure \
  --cookie-jar cookie.txt

curl '' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Origin:' \
  -H 'Referer:' \
  -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36' \
  --data-raw 'j_username=someusername+&j_password=adminmyapp' \\
  --cookie-jar cookie1.txt \
  --cookie cookie.txt

With cookie file I then can curl send my request as below :

curl '' --cookie cookie1.txt

Is there any way to do that in NetData? if yes can someone please help me with this? I wasn’t able to find similar examples.

Thank you

@ilyam8 I believe we could add support for cookie-based authentication.

If the go library can’t do exactly what’s described here (which is a bit tricky admittedly), we can split the task. As long as the library can send the cookie along, the step related to getting a valid cookie to use could be up to users to complete.


Each job accepts a cookie filename as authentication method (full path, readable by the netdata user).

Users ensure that they produce and refresh these filenames via cron jobs running before the cookies expire.

If a configured cookie isn’t there or the response fails, we log the issue.

Thats Wonderful @Christopher_Akritid1 ,

I was sure that somehow I can read the cookie, but yet, I can’t find an example or info how to read from go.d/httpcheck.conf , let’s say I have a file cookie.txt under /tmp or any other folder, how I will point my go.d collector to use that file as variable or use it as header within job?

I think we can try to add proper cookie auth support, but I need to set up a lab to test it. @pgro can you suggest an app with cookie auth so I can test it locally?

I was thinking of 1-step cookie auth, e.g. to add the following options:

  • cookie_auth_url
  • cookie_auth_method
  • cookie_auth_headers
  • cookie_auth_username
  • cookie_auth_password
  • cookie_auth_body
  • cookie_auth_renewal (renew cookie every N seconds)

If there are several steps it makes the configuration a bit more complex and I definitely need a lab to add the support.

A quick and simple solution is what @Christopher_Akritid1 suggested (cookie filename).

For now that’s perfectly fine, I just need a way to load the file to my job collector. Is there a way I can load the cookie from external file? Paypal also using same mechanism. I can generate the cookie with Curl and save it to a temp file, then I can use that file periodically in order to perform my http checks.

No, it is not possible now, but it is easy to add this functionality. Will you be able to help me to test that? You will need to compile go.d.plugin (needs golang installed or Docker) and replace the existing version with updated.

Yes sure , if you show me the way I can perform the test.

if you have golang installed:

git clone
cd go.d.plugin
git checkout http_cookie_file_option

sudo systemctl stop netdata
# adjust the path depends on your install prefix, could be `/opt/netdata/usr/libexec/netdata/plugins.d/go.d.plugin`
sudo cp bin/godplugin /usr/libexec/netdata/plugins.d/go.d.plugin

The updated version has cookie_file config option. Set it and start netdata (sudo systemctl start netdata) or better run the plugin in the debug mode.

Git clone ok , cd ok , but

:~/netdata-fixes/http-withcookie/go.d.plugin$ git checkout http_cookie_file_option
error: pathspec 'http_cookie_file_option' did not match any file(s) known to git
:~/netdata-fixes/http-withcookie/go.d.plugin$ ls -lah
total 221K
drwxr-xr-x 16 root root   26 Mar 20 19:40 .
drwxr-xr-x  3 root root    3 Mar 20 19:40 ..
drwxr-xr-x  2 root root    3 Mar 20 19:40 .devcontainer
-rw-r--r--  1 root root   44 Mar 20 19:40 .dockerignore
drwxr-xr-x  8 root root   13 Mar 20 19:44 .git
drwxr-xr-x  3 root root    6 Mar 20 19:40 .github
-rw-r--r--  1 root root   79 Mar 20 19:40 .gitignore
-rw-r--r--  1 root root  137 Mar 20 19:40 .yamllint.yml
-rw-r--r--  1 root root  452 Mar 20 19:40
-rw-r--r--  1 root root  35K Mar 20 19:40 LICENSE
-rw-r--r--  1 root root 1.4K Mar 20 19:40 Makefile
-rw-r--r--  1 root root  15K Mar 20 19:40
drwxr-xr-x  7 root root   12 Mar 20 19:40 agent
drwxr-xr-x  2 root root    3 Mar 20 19:40 cli
drwxr-xr-x  3 root root    3 Mar 20 19:40 cmd
drwxr-xr-x  3 root root    4 Mar 20 19:40 config
-rw-r--r--  1 root root 1.3K Mar 20 19:40 docker-compose.yml
drwxr-xr-x  2 root root    3 Mar 20 19:40 docs
drwxr-xr-x  3 root root    3 Mar 20 19:40 examples
-rw-r--r--  1 root root 8.5K Mar 20 19:40 go.mod
-rw-r--r--  1 root root 165K Mar 20 19:40 go.sum
drwxr-xr-x  2 root root    4 Mar 20 19:40 hack
drwxr-xr-x  2 root root   10 Mar 20 19:40 logger
drwxr-xr-x  8 root root    8 Mar 20 19:40 mocks
drwxr-xr-x 81 root root   82 Mar 20 19:40 modules
drwxr-xr-x 12 root root   13 Mar 20 19:40 pkg

@pgro My bad, should be git clone, updated the post

Great, moved on but stuck on compile,

go: downloading v2.9.5+incompatible
go: downloading v0.21.1
go: downloading v0.20.0
go: downloading v1.1.2-0.20210511102531-373a877eec92
go: downloading v0.19.5
go: downloading v0.7.7
go: downloading v3.9.0
go: downloading v0.0.0-20220304095617-2e8d9baf4ac2
go: downloading v1.0.0
agent/job/vnode/vnode.go:7:2: package io/fs is not in GOROOT (/usr/lib/go-1.15/src/io/fs)

Returned to prompt

go version
go version go1.15.15 linux/amd64

Edit: as far as I can see from go.mod file this need go.lang 1.20 at least :frowning:

You have a pretty old version of golang. You need either to compile on a server with new version or, if you have Docker engine installed, compile in a Docker container:

docker run -d --name golang -v $(pwd)/go.d.plugin:/go.d.plugin golang:1.20.2-bullseye sleep infinity
docker exec -it golang bash
cd /go.d.plugin/
git config --global --add /go.d.plugin
git pull
git checkout http_cookie_file_option

after it, you will have bin/godplugin in the go.d.plugin/ directory.

ok great, plugin seems ok now, let me test it and I will revert

I wasn’t able to make it work.

Here is my go.d/httpcheck.conf

  - name:
      User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36"
      body: 'j_username=user&j_password=pass'
    cookie_file: /tmp/vma_cookie.txt

Then under Debug I get

 DEBUG ] httpcheck[http://10_1_1_1/vma/mail/msgs] httpcheck.go:90 using accepted HTTP statuses [200]
[ WARN  ] httpcheck[http://10_1_1_1/vma/mail/msgs] collect.go:40 Get "": net/http: invalid header field value for "Cookie"
[ INFO  ] httpcheck[http://10_1_1_1/vma/mail/msgs] job.go:208 check success
[ INFO  ] httpcheck[http://10_1_1_1/vma/mail/msgs] job.go:228 started, data collection interval 5s
[ DEBUG ] run[manager] run.go:43 tick 0
[ WARN  ] httpcheck[http://10_1_1_1/vma/mail/msgs] collect.go:40 Get "": net/http: invalid header field value for "Cookie"
CHART 'netdata.execution_time_of_httpcheck_http://10_1_1_1/vma/mail/msgs' '' 'Execution time' 'ms' 'go.d' 'netdata.go_plugin_execution_time' 'line' '145000' '5' '' 'go.d' 'httpcheck

My cookie file is structured as below

# HTTP Cookie File
# Generated by Wget on 2023-03-20 21:38:07.
# Edit at your own risk.     FALSE   /vma    FALSE   0       JSESSIONID      23B508B767344EA167A4EB9B4DA4E59F

@ilyam8 Do I miss something?

Thank you

Wrong implementation, I will update it and let you know.

Ok, Thank so much ilyam

I pushed a commit, you need to git pull after cd go.d.plugin/. After it you can rebuild the plugin.

Thanx, but in general, what option should I use within my httpcheck.conf conf