HOWTO - HTTP EndPoint Collector with Cookie and user/pass

pgro · March 19, 2023, 7:08pm

Hi Everyone,

I got some requests and I need your help. We would like to add a few http endpoint collectors but in order to get the results I have first to pass the proper login. This is a 3 step process and from bash I can do that as below example:

curl 'http://10.1.1.1/myapp/index' \
  -H 'Referer: http://10.1.1.1/myapp/index' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0' \
  --insecure \
  --cookie-jar cookie.txt

curl 'http://10.1.1.1/myapp/j_security_check' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Origin: http://10.1.1.1' \
  -H 'Referer: http://10.1.1.1/myapp/j_security_check' \
  -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36' \
  --data-raw 'j_username=someusername+&j_password=adminmyapp' \\
  --cookie-jar cookie1.txt \
  --cookie cookie.txt

With cookie file I then can curl send my request as below :

curl 'http://10.1.1.1/myapp/myappfolder' --cookie cookie1.txt

Is there any way to do that in NetData? if yes can someone please help me with this? I wasn’t able to find similar examples.

Thank you
BR
PG

Christopher_Akritid1 · March 20, 2023, 2:41pm

@ilyam8 I believe we could add support for cookie-based authentication.

If the go library can’t do exactly what’s described here (which is a bit tricky admittedly), we can split the task. As long as the library can send the cookie along, the step related to getting a valid cookie to use could be up to users to complete.

e.g.

Each job accepts a cookie filename as authentication method (full path, readable by the netdata user).

Users ensure that they produce and refresh these filenames via cron jobs running before the cookies expire.

If a configured cookie isn’t there or the response fails, we log the issue.

pgro · March 20, 2023, 4:13pm

Thats Wonderful @Christopher_Akritid1 ,

I was sure that somehow I can read the cookie, but yet, I can’t find an example or info how to read from go.d/httpcheck.conf , let’s say I have a file cookie.txt under /tmp or any other folder, how I will point my go.d collector to use that file as variable or use it as header within job?

ilyam8 · March 20, 2023, 4:46pm

I think we can try to add proper cookie auth support, but I need to set up a lab to test it. @pgro can you suggest an app with cookie auth so I can test it locally?

ilyam8 · March 20, 2023, 5:52pm

I was thinking of 1-step cookie auth, e.g. to add the following options:

cookie_auth_url
cookie_auth_method
cookie_auth_headers
cookie_auth_username
cookie_auth_password
cookie_auth_body
cookie_auth_renewal (renew cookie every N seconds)

If there are several steps it makes the configuration a bit more complex and I definitely need a lab to add the support.

A quick and simple solution is what @Christopher_Akritid1 suggested (cookie filename).

pgro · March 20, 2023, 6:13pm

For now that’s perfectly fine, I just need a way to load the file to my job collector. Is there a way I can load the cookie from external file? Paypal also using same mechanism. I can generate the cookie with Curl and save it to a temp file, then I can use that file periodically in order to perform my http checks.

ilyam8 · March 20, 2023, 7:09pm

No, it is not possible now, but it is easy to add this functionality. Will you be able to help me to test that? You will need to compile go.d.plugin (needs golang installed or Docker) and replace the existing version with updated.

pgro · March 20, 2023, 7:22pm

Yes sure , if you show me the way I can perform the test.

ilyam8 · March 20, 2023, 7:28pm

if you have golang installed:

git clone https://github.com/ilyam8/go.d.plugin
cd go.d.plugin
git checkout http_cookie_file_option
hack/go-build.sh

sudo systemctl stop netdata
# adjust the path depends on your install prefix, could be `/opt/netdata/usr/libexec/netdata/plugins.d/go.d.plugin`
sudo cp bin/godplugin /usr/libexec/netdata/plugins.d/go.d.plugin

The updated version has cookie_file config option. Set it and start netdata (sudo systemctl start netdata) or better run the plugin in the debug mode.

pgro · March 20, 2023, 7:45pm

Git clone ok , cd ok , but

:~/netdata-fixes/http-withcookie/go.d.plugin$ git checkout http_cookie_file_option
error: pathspec 'http_cookie_file_option' did not match any file(s) known to git
:~/netdata-fixes/http-withcookie/go.d.plugin$ ls -lah
total 221K
drwxr-xr-x 16 root root   26 Mar 20 19:40 .
drwxr-xr-x  3 root root    3 Mar 20 19:40 ..
drwxr-xr-x  2 root root    3 Mar 20 19:40 .devcontainer
-rw-r--r--  1 root root   44 Mar 20 19:40 .dockerignore
drwxr-xr-x  8 root root   13 Mar 20 19:44 .git
drwxr-xr-x  3 root root    6 Mar 20 19:40 .github
-rw-r--r--  1 root root   79 Mar 20 19:40 .gitignore
-rw-r--r--  1 root root  137 Mar 20 19:40 .yamllint.yml
-rw-r--r--  1 root root  452 Mar 20 19:40 Dockerfile.dev
-rw-r--r--  1 root root  35K Mar 20 19:40 LICENSE
-rw-r--r--  1 root root 1.4K Mar 20 19:40 Makefile
-rw-r--r--  1 root root  15K Mar 20 19:40 README.md
drwxr-xr-x  7 root root   12 Mar 20 19:40 agent
drwxr-xr-x  2 root root    3 Mar 20 19:40 cli
drwxr-xr-x  3 root root    3 Mar 20 19:40 cmd
drwxr-xr-x  3 root root    4 Mar 20 19:40 config
-rw-r--r--  1 root root 1.3K Mar 20 19:40 docker-compose.yml
drwxr-xr-x  2 root root    3 Mar 20 19:40 docs
drwxr-xr-x  3 root root    3 Mar 20 19:40 examples
-rw-r--r--  1 root root 8.5K Mar 20 19:40 go.mod
-rw-r--r--  1 root root 165K Mar 20 19:40 go.sum
drwxr-xr-x  2 root root    4 Mar 20 19:40 hack
drwxr-xr-x  2 root root   10 Mar 20 19:40 logger
drwxr-xr-x  8 root root    8 Mar 20 19:40 mocks
drwxr-xr-x 81 root root   82 Mar 20 19:40 modules
drwxr-xr-x 12 root root   13 Mar 20 19:40 pkg
root@10.9.210.5:~/netdata-fixes/http-withcookie/go.d.plugin$

ilyam8 · March 20, 2023, 7:50pm

@pgro My bad, should be git clone https://github.com/ilyam8/go.d.plugin, updated the post

pgro · March 20, 2023, 7:54pm

Great, moved on but stuck on compile,

.
.
go: downloading github.com/emicklei/go-restful v2.9.5+incompatible
go: downloading github.com/go-openapi/swag v0.21.1
go: downloading github.com/go-openapi/jsonreference v0.20.0
go: downloading github.com/google/certificate-transparency-go v1.1.2-0.20210511102531-373a877eec92
go: downloading github.com/go-openapi/jsonpointer v0.19.5
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/emicklei/go-restful/v3 v3.9.0
go: downloading github.com/grafana/regexp v0.0.0-20220304095617-2e8d9baf4ac2
go: downloading github.com/josharian/intern v1.0.0
agent/job/vnode/vnode.go:7:2: package io/fs is not in GOROOT (/usr/lib/go-1.15/src/io/fs)

Returned to prompt

go version
go version go1.15.15 linux/amd64

Edit: as far as I can see from go.mod file this need go.lang 1.20 at least

ilyam8 · March 20, 2023, 8:07pm

You have a pretty old version of golang. You need either to compile on a server with new version or, if you have Docker engine installed, compile in a Docker container:

docker run -d --name golang -v $(pwd)/go.d.plugin:/go.d.plugin golang:1.20.2-bullseye sleep infinity
docker exec -it golang bash
cd /go.d.plugin/
git config --global --add safe.directory /go.d.plugin
git pull
git checkout http_cookie_file_option
hack/go-build.sh
exit

after it, you will have bin/godplugin in the go.d.plugin/ directory.

pgro · March 20, 2023, 8:25pm

ok great, plugin seems ok now, let me test it and I will revert

pgro · March 20, 2023, 9:52pm

I wasn’t able to make it work.

Here is my go.d/httpcheck.conf

  - name: http://10.1.1.1/vma/mail/msgs
    url: http://10.1.1.1/vma/mail/msgs
    headers:
      User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
      body: 'j_username=user&j_password=pass'
    cookie_file: /tmp/vma_cookie.txt

Then under Debug I get

 DEBUG ] httpcheck[http://10_1_1_1/vma/mail/msgs] httpcheck.go:90 using accepted HTTP statuses [200]
[ WARN  ] httpcheck[http://10_1_1_1/vma/mail/msgs] collect.go:40 Get "http://10.1.1.1/vma/mail/msgs": net/http: invalid header field value for "Cookie"
[ INFO  ] httpcheck[http://10_1_1_1/vma/mail/msgs] job.go:208 check success
[ INFO  ] httpcheck[http://10_1_1_1/vma/mail/msgs] job.go:228 started, data collection interval 5s
[ DEBUG ] run[manager] run.go:43 tick 0
[ WARN  ] httpcheck[http://10_1_1_1/vma/mail/msgs] collect.go:40 Get "http://10.1.1.1/vma/mail/msgs": net/http: invalid header field value for "Cookie"
CHART 'netdata.execution_time_of_httpcheck_http://10_1_1_1/vma/mail/msgs' '' 'Execution time' 'ms' 'go.d' 'netdata.go_plugin_execution_time' 'line' '145000' '5' '' 'go.d' 'httpcheck
'

My cookie file is structured as below

# HTTP Cookie File
# Generated by Wget on 2023-03-20 21:38:07.
# Edit at your own risk.

10.1.1.1.1     FALSE   /vma    FALSE   0       JSESSIONID      23B508B767344EA167A4EB9B4DA4E59F

pgro · March 21, 2023, 9:23am

@ilyam8 Do I miss something?

Thank you

ilyam8 · March 21, 2023, 12:43pm

Wrong implementation, I will update it and let you know.

pgro · March 21, 2023, 12:44pm

Ok, Thank so much ilyam

ilyam8 · March 21, 2023, 2:27pm

I pushed a commit, you need to git pull after cd go.d.plugin/. After it you can rebuild the plugin.

pgro · March 21, 2023, 2:33pm

Thanx, but in general, what option should I use within my httpcheck.conf conf

Topic		Replies	Views
How to use unix socket with zscores, changefinder, or anomalies modules Help agent , collectors , configuration	5	498	March 15, 2022
Pihole collector not working Help agent	2	588	December 31, 2022
Haproxy go.d configuration for stats page Help agent , collectors , configuration	3	342	March 2, 2023
Web_log.conf for nginx Help agent-collectors , agent	28	3029	November 8, 2021
Missing statistics like 'users' Help agent , cloud , collectors	2	351	June 6, 2023

HOWTO - HTTP EndPoint Collector with Cookie and user/pass

Related Topics